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I.  INTRODUCTION 


Inappropriate  and/or  duplicate  IT  systems  can  result  in  a  severe  drain  on 
resources.  The  development  of  low  utility  and  duplicate  systems  needs  to  be  identified  so 
that  system  development  resources  may  be  redirected  to  areas  with  higher  and  unique 
returns  (Valfer,  Kirby,  &  Schwarzbart,  1981).  Improving  systems  and  system  utilization 
can  result  in  long-term  benefits.  The  problem  is  there  is  not  a  measurement  methodology 
to  forecast  and  compare  systems.  The  purpose  of  this  research  is  to  provide  a 
methodology  to  forecast  and  compare  IT  investments,  which  can  be  used  in  IT 
investment  decisions. 

The  decision  maker  of  an  organization  must  be  able  to  monitor  how  IT  systems 
are  functioning  and  hold  program  managers  and  developers  accountable  for  improving 
efficiency,  timeliness,  and  accuracy  of  the  infonnation  being  gather  and  processed 
(Sniegowski,  2010).  The  Department  of  Defense  (DoD)  Chief  Information  Officer  (CIO) 
distributed  the  ten  point  implementation  plan  to  modernize  information  technology  (IT) 
within  the  DoD.  Included  in  those  ten  points  were  the  strategic  goals  of  strengthening  IT 
investments  and  reviewing  the  perfonnance  of  major  investments  (Tekai,  2012). 

Identifying  the  benefits  of  a  given  system  is  an  integral  part  of  determining  an  IT 
system’s  value  and  the  use  of  metrics  allows  for  the  measurement  of  that  value.  Return  on 
Investment  (ROI)  is  commonly  used  to  measure  the  value  of  respective  IT  investments 
and  as  a  comparative  metrics.  Benefits,  costs,  and  the  value  of  the  IT  over  time 
as  considered  when  measuring  ROI,  however  many  benefits  associated  with  particular 
IT  investments  are  difficult  to  translate  into  revenue  and  do  not  generalize  to  other  IT 
investments  making  it  difficult  to  calculate  an  ROI,  which  can  be  used  to  compare 
IT  investments. 

IT  management  is  not  purely  a  technical  issue  as  evidence  by  competing  IT  value 
measurements  that  attempt  to  rank  IT  systems  in  their  effectiveness,  efficiency,  and 
accountability  (Yang  &  Melitski,  2007).  The  challenge  for  management  is  to  have  an  IT 
strategy  that  improves  innovation,  flexibility,  efficiency,  and  visibility  of  the  underlying 
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processes  in  order  to  have  better  data  to  make  sound  decisions  (Housel  &  Bergin,  2013). 
Measuring  and  applying  volatility  to  an  IT  investment  serves  this  gap  and  can  be  used  in 
detennining  IT  strategic  decisions. 

A.  RETURN  ON  INVESTMENT  (ROI) 

Measuring  the  ROI  on  infonnation  systems  along  with  determining  the  factors 
and  conditions  that  affect  that  ROI  is  not  well  understood  (Roztocki  &  Weistroffer, 
2009).  While  some  authors  point  to  financial  ROI  in  the  form  of  money,  most  literature  is 
silent  on  measuring  information  system  ROI  (Fichenscher  &  Bakerman,  2011).  In  non¬ 
profit  organizations  such  as  the  Department  of  Defense  (DoD),  ROI  has  mostly  been  used 
to  measure  the  amount  of  savings  generated  against  an  alternative  for  each  investment 
dollar.  The  greater  the  ROI,  the  greater  the  advantage  is  to  the  organization  (NASA, 
2006),  however  using  that  metric  would  leave  a  lot  of  infonnation  systems  that  increase 
productivity  vice  save  costs  with  a  negative  ROI.  How  IT  investments  are  measured, 
along  with  financial  perfonnance,  determines  the  relationship  between  perfonnance  and 
those  IT  investments  (Lim,  Dehning,  Richardson,  &  Smith,  2011). 

Establishing  the  ROI  on  signals  intelligence  (SIGINT)  systems  through 
knowledge  value  added  (KVA)  was  shown  to  be  effective  from  the  previous  work  of 
Rios,  Lambeth,  and  Clapp.  The  next  iteration  of  research  is  to  configure  the  analysis  to 
determine  the  volatility  of  a  system  and  forecast  the  returns  to  detennine  if  a  system 
should  be  incrementally  upgraded  in  the  current  version  or  completely  moved  to  the  next 
version  so  that  resources  are  maximized.  By  applying  volatility  measurements  to  an 
infonnation  system,  the  relative  effectiveness  of  future  performance  can  be  forecasted. 
Forecasting  the  benefits  of  SIGINT  systems  assists  the  program  manager  in  detennining 
whether  to  pursue  an  incremental  or  complete  upgrade  based  on  volatility.  This  approach 
is  applicable  to  multiple  IT  systems  upgrades  or  replacements. 

B.  FACTORS 

Technology  drives  improvements  in  productivity  across  all  sectors  (Elias,  2000). 

However,  IT  investment  success  factors  are  still  not  fully  understood  (Roztocki  & 

Weistroffer,  2009).  One  such  factor  is  volatility.  A  survey  of  Chief  Financial  Officers 
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(CFOs)  listed  the  increased  volatility  and  risk  along  with  changes  in  strategy  as  a  top 
concern  for  the  future  business  environment  (Hosley,  2011).  The  volatility  of  a  portfolio 
or  asset  is  an  important  parameter  used  in  option  pricing,  risk  management,  and  asset 
allocation.  As  such,  there  are  several  approaches  available  to  forecast  the  future  volatility 
of  a  portfolio  (Ganesan  &  Yadav,  2007).  Risk  factors  are  especially  not  well  understood, 
but  applying  volatility  measurements  creates  awareness  of  the  risk  within  an  information 
system  (Roztocki  &  Weistroffer,  2009).  The  correlation  between  benefit  and  risk  factors 
provides  insight  to  the  success  of  IT  investment  decisions  (Zandi  &  Tavana,  2011).  Cost 
is  another  factor  with  high  cost  in  IT  investments  resulting  in  the  need  to  efficiently  and 
effectively  evaluate  IT  investments  for  organizations  (Zandi  &  Tavana,  2011).  The 
balance  between  system  stability  and  volatility  is  affected  by  many  such  factors  and  is  a 
significant  input  in  detennining  when  to  make  system  changes  (Kang,  2007). 
Management  and  monitoring  verify  productivity  gains  and  assist  in  detennining  the  value 
of  the  underlying  information  system  (Gholami,  2012). 

The  growing  complexity  of  IT  investments  makes  real-time  adjustments  a 
necessity  to  ensure  a  successful  information  system  (Kang,  2007).  Measuring  actual 
improved  capabilities  against  perfonnance  measures  provides  the  support  needed  to 
adjust  the  mix  of  portfolio  investments  in  order  to  maximize  the  returns  to  DoD  (DoD, 
2005).  Having  a  measurement  methodology  that  provides  visibility  and  agility  to  respond 
to  changes  in  the  environment  is  critical  (Hosley,  2011).  In  the  financial  sector,  the 
Volatility  Index  (VIX)  provides  that  mechanism  (D’Anne,  2012),  and  a  similar  approach 
could  be  used  toward  information  systems. 
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C.  RESEARCH  QUESTIONS 

There  are  three  research  questions  reviewed  as  it  relates  to  volatility 
measurements. 

1 .  Do  volatility  measurements  provide  a  capability  to  forecast  future  perfonnance 
of  the  system? 

2.  Do  volatility  measurements  provide  timing  options  for  the  implementation  of 
incremental  or  new  information  systems? 

3.  Can  beta  be  detennined  within  a  portfolio  of  systems  based  on  volatility 
measurements? 

The  paper  is  organized  as  follows.  Chapter  II  provides  an  overview  of  literature 
on  the  many  factors  that  affect  the  perceived  success  of  an  IT  system  with  specific  focus 
on  volatility.  Chapter  III  provides  the  methodology  of  this  research  to  answer  the  research 
questions.  Chapter  IV  reviews  the  analysis  of  the  data.  Chapter  V  draws  conclusion  of  the 
analysis  in  chapter  IV  and  answers  the  research  questions  along  with  providing 
recommendations  for  implementation  and  future  research. 
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II.  LITERATURE  REVIEW 


Information  is  a  strategic  asset  and  transforming  it  to  knowledge  is  essential  (DoD 
CIO,  2012).  Infonnation  systems  provide  that  transfonnation  from  infonnation  to 
knowledge  as  well  as  obtaining  efficiencies  in  delivering  support  (Olson  &  Wu,  2011). 
The  DoD  Infonnation  Enterprise  Architecture  (IEA)  provides  the  strategic  architecture 
from  which  to  govern  and  integrate  infonnation  systems  (DoD  CIO,  2012).  It  is  desired 
that  every  information  system  project  should  execute  within  budget  and  on  time  and  any 
risks  are  identified  early  (A1  Kattan,  A1  Haddad,  &  A1  Ali,  2011). 

The  DoD  IEA  provides  the  governance  from  which  to  operate  the  IT  landscape. 
Decision  makers  prioritize  the  needed  capabilities  by  detennining  the  gaps  between 
existing  and  required  IT  capabilities  and  plan  the  appropriate  investment  options  and 
resources  to  those  gaps.  They  then  use  those  capabilities  to  establish  the  criteria  and 
performance  metrics  for  future  investment  decision  making  as  well  as  grading  of  past 
investments  decisions  (DoD  CIO,  2012).  Volatility  measurements  are  used  frequently  in 
the  financial  management  sector  and  could  be  used  to  provide  additional  information  to 
aid  in  the  investment  decision  making  process.  The  DoD  IEA  provides  capability 
descriptions  that  allow  the  decision  makers  to  identify,  evaluate,  and  compare  different 
investments  to  a  baseline  and  also  identify  the  investment  risks  (DoD  CIO,  2012). 
Volatility  measurements  in  the  comparison  of  a  baseline  allows  for  the  formulation  of  a 
beta.  Beta  is  a  common  investment  metric  that  can  also  be  used  to  forecast  future 
performance  and  identify  the  risks  associated  with  a  particular  information  system. 

“In  reality,  IT  project  managers  face  considerable  uncertainty  in  determining  the 
likely  extent  of  any  risk  factor  identified  as  a  potential  threat,  and  therefore,  uncertainty 
about  possible  solutions  in  terms  of  their  cost  and  effectiveness”  (Taylor,  Artman,  & 
Woelfer,  2012,  p.  19). 

This  section  reviews  the  governance  of  IT  systems,  the  factors  that  affect 
performance,  IT  integration,  metrics,  volatility,  risk,  beta,  and  forecasting  of  IT 
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performance.  Within  each  subsection,  the  benefits  for  use  of  volatility  measurements  are 
discussed  as  it  pertains  to  the  different  factors. 


A.  GOVERNANCE  OF  IT  SYSTEMS 

The  proper  governance  of  an  IT  system  is  essential  to  detennine  if  a  system  is 
providing  a  positive  return  on  investment.  Two  broad  categories  that  factor  in  governance 
are  the  strategy  and  the  measurements.  Prior  to  measuring  volatility  of  any  measurement, 
it  is  imperative  to  detennine  the  strategy  that  precipitated  the  IT  investment.  The 
Department  of  Defense  organizational  execution  plans  (OEPs)  provide  the  mission  that 
the  IT  investments  are  to  effect  (DCMO,  2013).  Once  the  business  area  is  determined,  an 
appropriate  metric  is  used  to  measure  the  performance  of  the  IT  system  along  with 
managerial  performance  (Merikas,  Merikas,  &  Sonos,  2005).  The  metrics,  in  agreement 
with  the  overall  strategy,  then  determine  the  options  in  the  continuation  of  IT  services 
and  infrastructure  in  other  functional  areas  (DCMO,  2013). 

Strategy  guides  the  governance  model  to  organize  and  control  IT  projects 
(Gholami,  2012).  The  governance  model  can  also  be  referred  to  as  investment 
management  where  the  functional  strategies  are  aligned  with  the  overall  Strategic 
Management  Plan  (SMP)  and  along  with  the  OEPs  determine  the  factors  that  drive  value 
in  the  IT  investment  portfolio  (DCMO,  2013).  It  is  only  when  the  business  processes  and 
IT  strategy  are  closely  coupled  that  performance  improves  (Kang,  2007).  Capturing  the 
metrics  to  detennine  the  maturity  and  changes  in  strategy  is  crucial  in  ensuring  that 
decision  makers  are  able  to  respond  to  those  changes.  The  directive  to  manage  IT  systems 
as  a  portfolio  through  IT  portfolio  management  (DoD,  2005)  further  increases  the  need  to 
mitigate  negative  reactions  to  fluctuations  in  the  IT  systems  along  with  determining  the 
risk  tolerance  for  a  system  (Gilliam,  Chatterjee,  &  Grable,  2010). 

The  use  of  measurements  is  essential  in  governance.  Unforeseen  external 
circumstances  generally  demand  system  changes.  Data  analysis  identifies  those  demands 
of  change  by  measuring  several  objectives  and  provides  an  opportunity  for  decision 
makers  to  weigh  the  chaos  of  disruption  versus  that  of  the  benefits  of  change  in  the 
system  (Kang,  2007).  Historical  data,  especially  of  the  standard  deviation,  assists  in  the 
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governance  of  the  IT  systems  and  provide  metrics  from  which  to  compare  risks  and 
opportunity  (Merikas  et  ah,  2005).  The  volatility  measurements  manage  expectations  of 
the  IT  portfolio  along  with  financial  risk  tolerance  (Gilliam  et  ah,  2010).  The  governance 
of  the  IT  systems  has  to  be  effective  and  the  IT  strategy  aligned  with  the  overall  strategy 
and  appropriate  measurement  of  factors  in  order  for  the  IT  systems  to  be  measured  for 
effectiveness  and  efficiency. 

IT  changes  are  a  balancing  act  between  system  volatility  and  system  stability 
(Kang,  2007).  As  the  environment  changes  and  strategies  mature,  governance  needs  to  be 
refined  (DCMO,  2013).  The  decision  in  making  changes  in  policy  and/or  IT  systems 
needs  to  have  as  a  factor  the  moderation  of  output  volatility  (Chichilnisky  &  Gorbachev, 
2004).  Investment  portfolios  are  measured  adequately  through  statistics,  specifically  the 
mean  and  standard  deviation,  which  is  commonly  used  to  describe  the  volatility  of  a 
portfolio  (Merikas  et  ah,  2005). 

Every  IT  system  needs  to  be  evaluated  with  all  available  measures.  However, 
program  managers  should  be  wary  of  operational  concerns  for  the  IT  system  overriding 
the  prevailing  IT  strategy  (Housel  &  Bergin,  2013).  The  primary  intent  is  a  strategy  and 
management  approach  that  encompasses  the  enterprise  and  provides  for  common 
integrated  technical  infrastructure  and  services  (Yang  &  Melitski,  2007).  Managers  need 
to  ensure  that  IT  strategy  fits  the  overall  business  objective,  that  they  have  a  thorough 
understanding  of  the  business  needs  that  the  IT  system  addresses,  and  apply  the  IT  to 
implement  new  ways  of  doing  business  (Stikeleather,  2013).  An  IT  strategy  is  a 
comprehensive  plan  that  gives  a  roadmap  on  how  IT  will  accomplish  the  objectives  and 
principles  of  a  particular  organization  (Housel  &  Bergin,  2013).  Strategic  values  such  as 
efficiency  and  accountability  should  be  addressed  in  the  IT  strategy,  especially  for 
government  (Yang  &  Melitski,  2007).  Metrics  need  to  track  and  record  the  effect  of  IT  on 
productivity  along  with  the  benefits  to  the  operators  (Housel  &  Mun,  2013). 
Accomplishing  the  strategic  goals  of  the  DoD  CIO  for  strengthening  IT  investments  and 
review  performance  of  IT  investments  (Tekai,  2012),  can  be  accomplished  with  a 
volatility  measurement  of  any  appropriate  metric.  Measuring  the  volatility  of  metrics  over 
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time  will  provide  options  for  the  IT  managers  along  with  real-time  tracking  of  the  value 
of  that  particular  IT  system. 

Analysis  of  the  metrics  over  time  can  categorize  the  returns  in  terms  of  volatility 
with  the  greater  standard  deviation  resulting  in  higher  volatility.  Once  the  volatility  is 
determined,  the  returns  of  a  new  system  can  be  forecasted  and  tracked  against  that 
forecast  to  determine  the  increase  (or  decrease)  in  benefits  for  that  system.  For  example, 
evaluating  the  Cryptologic  Carry-on  Program  (CCOP)  would  start  with  assigning  a 
surrogate  for  revenue  such  as  unit  cost  ($/signal).  A  forecast  for  the  system  is  then 
generated  based  on  the  historic  volatility  as  well  as  the  volatility  of  a  previous  system. 
The  operating  costs  for  both  the  present  and  previous  system  are  applied  to  the  metric. 
The  metric  can  then  be  tracked  against  the  forecast  to  determine  the  gain  or  loss. 
Adjustments  can  be  made  for  unique  risks  between  the  systems  and  then  the  resulting 
forecasts  can  be  compared  to  alternative  systems.  To  ensure  the  greatest  return  any 
methodology  should  capture  the  changes  in  benefits  as  well  as  costs  while  capturing  the 
operational  benefits,  unit  cost,  and  variance  (Ford,  Housel,  &  Mun,  2009).  Measuring  the 
volatility  of  the  metrics  in  a  given  IT  system  provides  an  additional  tool  and  methodology 
in  making  strategic  decisions. 

Lack  of  infonnation  in  a  valuation  model  makes  it  more  difficult  for  managers  to 
make  informed  decisions  on  an  IT  system  and  properly  identify  options  (Copeland  & 
Tufano,  2004).  The  ultimate  goal  is  to  maximize  the  benefits  of  an  IT  system  while 
limiting  cost  and  risk,  especially  in  a  cost-constrained  environment  (Ford  et.  ah,  2009). 
Providing  volatility  measurements  increases  the  infonnation  that  a  manager  has  to 
properly  value  options.  The  value  assigned  to  IT  systems  with  certain  assumptions  can 
then  be  mapped  out  on  a  decision  tree  visibly  showing  the  value  of  a  system  at  a 
particular  time  (Copeland  &  Tufano,  2004). 

The  custom-built  decision  tree  based  on  forecasts  provides  the  options  from 

which  the  manager  can  monitor  and  make  decisions  for  the  IT  system  (Copeland  & 

Tufano,  2004).  The  decision  tree  provides  the  foundation  from  which  a  real-time  system 

can  compare  the  relative  volatility  measurements  and  provide  the  strategic  goals  of  agility 

and  interoperability.  With  a  more  accurate  model,  managers  are  more  likely  to  make 

8 


decisions  in  a  rational  and  timely  manner  (Copeland  &  Tufano,  2004).  Volatility 
measurements  should  decrease  as  technology  is  adopted  and  matures  which  can  then  be 
compared  to  previous  systems  to  detennine  the  real  impact  that  the  particular  system  is 
providing.  Aligning  the  budgeting  and  planning  systems  to  the  overall  IT  strategy  along 
with  the  individual  decision  trees  should  also  improve  the  timeliness  of  decisions 
regarding  IT  systems  (Copeland  &  Tufano,  2004). 

IT  systems  go  through  four  phases:  research  and  development,  ascent,  maturity, 
and  decline  (Housel  &  Bergin,  2013).  The  volatility  measurement  for  a  given  metric 
should  be  high  during  the  research  and  development  phases,  decreasing  as  it  is  in  the 
ascent  phase,  minimize  at  maturity  and  then  start  increasing  as  it  enters  the  decline  phase. 
Changes  in  the  volatility  measurement  allows  for  the  explicit  review  of  nodes  in  a 
decision  tree  and  its  alignment  with  the  forecasting  as  well  as  options  (Copeland  & 
Tufano,  2004).  Organized  appropriately,  the  IT  systems  will  develop  new  and  more 
effective  roles  within  the  enterprise  ecosystem  (Stikeleather,  2013).  As  an  IT  system 
matures,  it  will  progress  through  five  stages:  bleeding  edge,  leading  edge,  state  of  the  art, 
dated,  and  obsolete  (Housel  &  Bergin,  2013).  The  maturity  stages  can  be  aligned  with  the 
growth  phases  that  a  volatility  measurement  could  identify.  In  the  maturity  model,  the 
volatility  would  start  high,  minimize  at  state  of  the  art,  and  then  increase  until  high  again 
at  the  obsolete  stage.  Providing  a  volatility  measurement  allows  for  a  more  accurate 
forecasting  of  an  IT  system  and  provides  additional  infonnation  from  which  to  decide  on 
options,  and  the  framework  for  a  real-time  monitoring  system.  Better  decisions  on 
options  along  with  the  ability  to  track  perfonnance  will  strengthen  IT  investments  and  the 
perfonnance  of  those  systems. 

B.  FACTORS  AFFECTING  IT  EFFECTIVENESS  AND  EFFICIENCY 

Governance  provides  the  management  and  overall  environment  from  which  the 
information  system  operates  and  is  a  primary  factor  in  detennining  the  success  of  that 
information  system.  There  are  many  other  factors  that  affect  IT  investment  performance 
including  organizational  size,  fit,  and  system  characteristics  (Roztocki  &  Weistroffer, 
2009).  The  resources  allocated  to  the  information  system  along  with  the  technical 
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feasibility  are  also  of  great  importance  (Kang,  2007).  These  factors  are  reviewed  and 
assessed  into  broad  categories  of  costs,  dynamics,  perfonnance  criteria  and  integration. 

Cost  appears  to  be  a  constant  factor  in  all  systems,  IT  or  otherwise,  as  it  is  a 
constraining  resource.  Cost  has  the  benefit  of  having  a  large  amount  of  data  and  systems 
in  place  that  can  monitor  and  trace  cost  to  specific  services  and  products.  However, 
undependable  cost  estimates  can  lead  to  flawed  strategic  and  operational  decisions 
(Roztocki  &  Weistroffer,  2009),  which  discussed  earlier  would  significantly  hamper  the 
effectiveness  of  the  information  system.  Government  appears  to  focus  on  reducing  costs, 
implementing  Enterprise  Resource  Planning  (ERP)  to  standardize  systems  while  also 
outsourcing  IT  staffs  in  order  to  avoid  recruiting,  developing,  and  retaining  such 
personnel  (Olson  &  Wu,  2011).  Effective  cost  management  systems  provide  management 
a  factor  that  they  can  control  which  even  further  highlights  the  focus  on  costs.  The  logical 
conclusion  is  that  an  effective  cost  management  system  would  lead  to  fewer  mistakes  in 
IT  investments  (Roztocki  &  Weistroffer,  2009).  However,  information  system  risk  is  not 
only  costs,  but  disruptions,  hidden  costs,  future  upgrades,  etc.  (Olson  &  Wu,  2011). 
Although  cost  is  a  primary  factor  in  the  efficiency  of  an  information  system,  the  focus  on 
costs  appears  to  result  in  little  to  no  attention  on  factors  that  are  just  as  important  to  a 
system  such  success  such  as  system  dynamics,  performance  criteria  and  integration. 

An  incomplete  understanding  of  the  system  dynamics  with  the  infonnation  system 
and  the  structural  relationships  that  it  operates  within,  leads  to  volatility  in  forecast  which 
affects  future  costs  (Scher  &  Koomey,  2011).  The  standardization  of  systems  and 
measurements  has  increased  the  risk  of  missing  the  dynamics  of  the  structural 
relationship  and  thereby  increases  the  overall  risk  of  the  information  system  and  forecasts 
made  (Long,  2010).  Working  through  the  dynamics  of  the  environment  is  essential  to 
capture  best  practices,  pace  of  change,  and  allows  for  a  larger  source  of  data.  Institutions 
and  the  people  within  adapt  to  new  structural  relationship  and  increase  productivity 
should  the  relationship  be  fully  understood  (Scher  &  Koomey,  2011).  Forecasting  is  not 
new  to  business  or  the  DoD  and  with  the  additional  data;  those  forecasts  become  more 
accurate  and  allow  decision  makers  to  make  more  infonned  decisions  with  empirical  data 
(Hosley,  2011).  Volatility  measurements  of  performance  metrics  provide  a  visible 
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indicator  of  when  adaptation  of  system  dynamics  for  the  information  system  has  occurred 
and  can  also  indicate  the  lack  of  adaptation. 

There  are  many  different  performance  criteria  relevant  to  decisions  and  outcomes 
for  information  systems  (Roztocki  &  Weistroffer,  2009).  The  DoD  IEA  determines 
compliance  in  achieving  effectiveness  and  efficiency  goals  (DoD  CIO,  2012).  The 
perfonnance  criteria  selected  is  highly  significant  to  the  information  system’s  perceived 
perfonnance  (Kang,  2007).  For  example,  the  unit  cost  for  a  production  criteria  should 
decrease  as  production  increases,  becoming  more  efficient  in  the  utilization  of  resources 
(Chichilnisky  &  Gorbachev,  2004).  However,  a  perfonnance  criterion  for  perceived 
usefulness  and  user  satisfaction  is  critical  as  those  two  factors  are  indicative  of  the 
continual  use  of  a  given  information  system.  A  misalignment  of  perfonnance  criteria  with 
the  overall  strategy  and  goals  could  result  in  a  constantly  changing  network  environment 
and  a  disruption  to  the  overall  system  (Kang,  2007).  A  lot  of  focus  is  on  the  perfonnance 
criteria  of  efficiency,  but  operating  at  peak  effectiveness  is  just  as  important  if  not  more 
so  (Hosley,  2011).  Whatever  the  performance  criteria,  volatility  measurements  of  those 
criteria  are  taken  leading  to  insights  to  the  outcomes  and  decisions  of  the  information 
system  being  measured. 

There  are  many  factors  that  affect  perfonnance  such  as  security,  reliability, 
scalability,  availability,  monitoring  and  management,  and  all  of  these  factors  need  to  be 
integrated  (Olson  &  Wu,  2011).  The  many  factors  make  for  an  extremely  complex 
system  that  supports  the  business  processes  (Kang,  2007).  Integration  has  to  be  focal 
point  in  the  implementation  of  any  infonnation  system  and  has  led  to  technologies  such 
as  enterprise  application  integration  (EAI)  (Roztocki  &  Weistroffer,  2009).  Another 
source  of  complexity  is  the  network  connections  that  infonnation  system  makes  to  other 
systems  (Kang,  2007).  This  complexity  leads  to  a  lack  of  structural  constancy  in  the  early 
phases  of  a  system  as  changes  are  constantly  made  to  integrate  systems.  Physical  systems 
such  as  production  lines  are  an  example  of  structural  constancy  but  information  systems 
generally  do  not  exhibit  the  constancy  needed  to  accurately  forecast.  Structural  constancy 
is  necessary  for  forecasting  perfonnance  accurately  (Scher  &  Koomey,  2011).  Once  the 
information  system  is  integrated  it  is  reasonable  to  expect  that  existing  resources  are  used 

11 


more  efficiently  as  data  allows  for  more  informed  and  better  decisions  (Roztocki  & 
Weistroffer,  2009).  The  goal  then  becomes  to  be  more  responsive  and  proactive,  and  that 
requires  increased  certainty  in  volatile  times  which  comes  from  an  integrated  system 
(Hosley,  2011).  Volatility  measurements  would  give  additional  data  that  could  indicate 
the  effectiveness  of  the  integration  of  the  information  system  along  with  determining  the 
volatility  environment. 

Costs,  system  dynamics,  perfonnance  criteria  and  integration  are  the  critical  few 
of  the  many  factors  that  influence  IT  investment  decisions.  Volatility  measurements 
allow  for  further  investigations  into  the  influence  those  factors  have  on  the  perfonnance 
of  infonnation  systems  (Zandi  &  Tavana,  2011).  In  order  to  fully  benefit  from 
perfonnance  measurements,  the  systems  need  to  be  observable,  structurally  consistent, 
constant  across  variations,  and  pennit  collection  of  large  depth  of  data  (Scher  &  Koomey, 
2011).  And  as  experience  is  gained  with  the  infonnation  system,  the  application  use  and 
therefore  benefit  should  increase  (Kang,  2007).  Volatility  measurements  in  infonnation 
systems  may  detennine  the  drivers  that  should  influence  decisions  along  with  the  timing 
of  those  decisions. 

C.  INTEGRATION 

Assumptions  and  risks  can  undennine  the  most  adroit  implementation  of  an  IT 
system.  Volatility  measurements  provide  a  tool  from  which  assumptions  and  risks  are 
lessened.  Interoperability  allows  for  the  sharing  of  data  and  should  be  built  into  the 
strategy  in  order  to  get  the  benefits  desired  from  an  IT  investment.  Demands  of  the 
external  environment  require  the  ability  to  adjust  practices  quickly,  which  would  be  an 
agility  component  (Guertin  et.  ah,  2012).  An  interoperable  and  agile  system  along  with 
volatility  measurements  can  positively  affect  the  assumptions  and  risks  made  within  an 
IT  system. 

The  current  IT  environment  relies  on  integrated  rather  than  stand-alone  systems 
that  interact  with  multiple  software  and  hardware  components  (Kang,  2007). 
Interoperability  allows  for  the  sharing  of  data  and  should  be  built  into  the  strategy  in 
order  to  get  the  benefits  desired  from  an  IT  investment.  There  is  an  ideal  balance  between 
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innovative  ideas  and  efficient  focused  management  (Guertin  et.  al.,  2012)  and  with  an 
interoperable  environment  with  substantial  data  collection  that  balance  can  quickly  be 
located.  First,  the  proper  enterprise  architecture  for  a  system  has  to  be  detennined.  For 
example,  a  distributed  system  where  infonnation  is  analyzed  centrally  appears  to  be  best 
served  with  a  client-server  architecture  where  one  computer  acts  as  a  server  and  others  as 
clients  (Chang  &  West,  2006).  Interoperability  is  a  challenge  in  inherently  distributed  and 
horizontal  systems  and  in  order  to  acquire  fast  and  effective  responses,  interoperability 
has  to  be  built  into  the  structure  (Kasunic,  2002).  Second,  boundaries  between  entities  is 
disappearing  both  horizontally  and  vertically,  which  demands  interoperability  in  the 
enterprise  IT  system  (Stikeleather,  2013).  The  interoperability  performance  can  be 
measured  with  the  Levels  of  Information  Systems  Interoperability  (LISI),  which  proposes 
four  sets  of  interoperability  measures:  technical  compliance,  system  interoperability, 
operational  interoperability,  and  organizational  and  cultural  (Kasunic,  2002).  The 
horizontal  integration  and  interoperability  in  order  to  share  infonnation  is  especially  a 
daunting  challenge  which  if  solved  could  unleash  substantial  returns  in  an  IT  system 
(Yang  &  Melitski,  2007). 

The  digital  ecosystem  is  a  demand-driven  environment  where  systems  need  to  be 
proactive  and  responsive  (Chang  &  West,  2006).  Network  performance  and  integration 
are  challenges  in  developing  an  agile  system  (Kang,  2007).  The  digital  ecosystem  is  ever 
changing  with  bandwidth  growing  at  least  three  times  faster  than  computer  power 
(Gilder’s  law)  and  magnetic  storage  costs  reducing  by  half  every  18  months  (Shugart’s 
law),  which  when  combined  with  the  value  of  a  network  being  the  square  of  the  number 
of  users  of  that  system  (Metcalfe’s  law)  (Housel  &  Bergin,  2013),  the  agility  within  a 
system  and  organization  becomes  crucial.  Education  and  training  can  increase 
organizational  agility  (Guertin  et.  al.,  2012);  however,  if  it  is  not  inherent  in  the  IT 
system,  then  the  organization  may  still  fail  to  meet  its  objectives.  There  are  often 
conflicting  preferences  and  values  regarding  IT  investments  and  all  are  legitimate  and 
important,  which  requires  an  agile  system  to  respond  to  the  fluid  environment  (Yang  & 
Melitski,  2007).  An  agile  system  would  adjust  to  the  demands  of  the  external 
environment  (Guertin  et.  al.,  2012). 
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Assumptions  are  made  throughout  the  technology  life  cycle  as  models  are  used 
that  give  a  simplified  representations  of  the  environment,  usage,  learning  curves,  etc. 
(Copeland  &  Tufano,  2004).  The  assumptions  are  then  used  to  balance  the  value  as  it 
relates  to  efficiency  and  agility  in  order  to  provide  IT  systems  with  interoperability  and 
agility  (Stikeleather,  2013).  The  value  of  the  underlying  system  may  not  be  clear; 
therefore,  past  performance  sets  an  estimated  value  (Copeland  &  Tufano,  2004). 
Assumptions  could  affect  the  answer  to  questions  such  as  should  investments  be  made 
toward  personnel  or  new  technology  and  what  sub-processes  should  receive  further 
investments.  Measuring  the  volatility  for  metrics  that  measure  critical  objectives  would 
give  insight  as  to  which  IT  system  or  sub-process  is  truly  having  the  desired  effect  and 
thereby  reducing  the  assumptions  made. 

Integrated  risk  management  is  multiple  step  process  that  identifies,  predicts, 
models,  analyzes,  mitigates,  hedges,  diversifies,  and  manages  risk  (Ford  et.  al.,  2009). 
Risk  identification  is  closely  tied  to  assumptions  made  in  assessing,  selecting,  and 
implementing  an  IT  system.  One  such  risk  would  be  a  lacking  a  strategic  perspective, 
focusing  instead  on  budgetary  and  operational  efficiencies  (Yang  &  Melitski,  2007). 
Other  factors  may  include  organizational  design  or  socio-technical  work  design  where  a 
mismatch  could  produce  a  potential  for  risk.  Risk  prediction  is  the  projection  of  an  IT 
project,  forecasting  the  effect  of  the  project  over  time.  The  risk  model  gives  the  financials 
on  a  given  project  under  certain  circumstances.  Risk  analysis  is  the  methodology  of 
running  simulations  to  determine  the  actual  risk  involved.  Risk  mitigation  is  the 
comparison  of  alternatives  and  the  framing  of  options.  Risk  hedging  is  the  building  of 
options  within  the  system.  Another  risk  is  the  data,  as  incomplete  adoption  and  non¬ 
adoption  of  IT  systems  may  not  be  available  to  make  the  comparison  with  the  adopted 
IT  systems  in  order  to  determine  the  differences  in  metrics  (Sahin,  2006).  Risk 
diversification  is  optimization  across  several  systems  and  finally,  risk  management  deals 
with  the  tracking  and  updates  as  the  system  progresses  as  it  pertains  to  risk.  Risk 
management  may  presume  a  stable,  predictable,  controllable  environment,  which  may  not 
be  the  case  especially  in  the  digital  ecosystem;  therefore  agility  should  be  identified  and 
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measured  (Guertin  et.  al.,  2012).  In  each  case,  measuring  volatility  lessens  the  risk  by 
providing  additional  infonnation  and  identifying  key  points  in  the  system’s  life  cycle. 

Although  assumptions  and  risks  will  continue  to  be  part  of  the  IT  investment 
environment,  volatility  measurements  provide  a  tool  to  lessen  those  assumptions  and 
risks  leading  to  a  more  accurate  forecast  of  performance.  Measuring  the  volatility  can 
also  assist  in  determining  the  adoption  of  the  technology.  Technology  adoption  from  the 
user  prospective  is  broken  up  into  five  groups,  the  innovators,  early  adopters,  early 
majority,  late  majority,  and  laggards  (Housel  &  Bergin,  2013).  The  volatility 
measurement  will  be  high  in  the  early  adoption  and  will  start  to  decrease  noticeably  as 
the  early  majority  begins  to  adopt  the  technology  with  the  volatility  smoothing  out  near  a 
low  as  the  laggards  begins  to  finally  adopt  the  technology.  Integration  and  agility  are 
critical  success  factors  for  an  IT  system,  increasing  communication  between  stakeholders 
and  quickly  aligning  with  any  changes  in  objectives  (Yang  &  Melitski,  2007).  An  IT 
system  with  integration  and  agility  along  with  an  accurate  forecast,  with  the  help  of  a 
volatility  measurement,  will  strengthen  IT  investments  and  the  performance  review. 

D.  METRICS 

There  is  a  move  to  quantify  data  drivers  and  include  them  into  an  Enterprise 
Performance  Management  (EPM)  program  (Hosley,  2011).  Those  data  drivers  are  the 
many  different  metrics  that  are  used  to  assist  decision-making.  This  section  reviews  the 
decision-making  metric  needs,  some  of  the  different  metrics  available,  and  the  challenges 
in  using  those  metrics  to  measure  the  perfonnance  of  an  infonnation  system. 

Quantifiable  performance  measures  are  used  to  manage  and  monitor  IT 
investments.  Those  measures  allow  for  the  grading  and  tracking  of  investment  decisions 
and  allow  for  an  evaluation  to  modify,  continue,  or  tenninate  investments  based  on  those 
outcome-based  measures.  The  perfonnance  measures  also  allow  for  a  determination  of 
the  investment  risk  for  an  infonnation  system  (DoD  CIO,  2012).  Understanding  the 
performance  measures  and  the  infonnation  that  it  presents  is  crucial  in  order  to  be  more 
proactive  and  make  better  decisions  (Hosley,  2011).  Fully  understanding  the  information 
allows  for  an  analysis  to  determine  the  progress  made  in  filling  a  capability  and  assists  in 
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identifying  gaps  as  described  in  an  Infonnation  Support  Plan  (ISP).  Once  the  gaps  are 
identified  then  it  is  detennined  how  available  funds  are  spent  to  close  those  gaps.  An 
integrated  information  system  provides  the  information  needed  to  decision-makers  that 
allow  them  to  take  action  on  issues  and  make  better  infonned  decisions  (DoD  CIO, 

2012) .  Volatility  measurements  provide  an  additional  data  point  for  decision-makers  to 
take  action  potentially  providing  the  critical  information  of  when  to  take  action. 

Portfolio  managers  evaluate  the  IT  investment  periodically  to  ensure  that  the 
system  is  providing  actual  support  (DoD  CIO,  2012).  There  are  several  IT  evaluation 
methods  including  net  present  value  (NPV),  ROI,  infonnation  economics,  cost  benefit 
analysis,  and  return  on  management  that  provides  the  framework  to  quantify  the  benefits 
and  risks  (Zandi  &  Tavana,  2011).  The  evaluation  contains  measureable  outcomes  and 
targets  as  determined  by  the  functional  strategies  of  the  information  system  (DCMO, 

2013) .  The  ISPs  are  used  to  determine  the  progress  in  achieving  the  stated  capabilities 
within  the  ISP  (DoD  CIO,  2012).  The  evaluation  frameworks  that  quantify  the  goals 
should  be  part  of  the  functional  strategy  and  aligned  with  the  overall  objectives. 

The  desired  end  state  of  an  information  system  should  able  to  be  detennined  by 
looking  at  the  measurable  and  observable  goals  (DCMO,  2013).  Reduction  of  costs  will 
generally  be  a  primary  goal  which  is  accomplished  through  the  reduction  of  IT  staff, 
better  inventory  control,  or  duplicate  system  elimination  (Olson  &  Wu,  2011).  Other 
metrics  could  include  benefits  such  as  better  output  and  increased  skills  or  could  include 
indirect  costs  such  as  decreased  status,  unpleasant  routines,  or  lost  time.  The  consistency 
of  the  benefit/cost  ratio  is  desirable  so  that  the  ratio  does  not  vary  among  different 
managers  or  over  time  (Kang,  2007).  Cost  is  an  important  metric  and  is  essential  to 
determine  ROI,  but  there  are  other  metrics  as  important  but  may  be  difficult  to  fit  into  an 
evaluation  framework  such  as  product  functionality  and  quality,  implementation  and 
interface  speed,  price,  reliability,  scalability,  availability,  customer  service,  security, 
service  level  monitoring  and  management  (Olson  &  Wu,  2011).  With  all  the  different 
metrics  available  it  is  essential  for  the  managers  to  ensure  that  the  performance  measures 
are  directly  tied  to  the  strategy  (DCMO,  2013).  Volatility  measurements  work  across  the 
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entire  spectrum  of  quantifiable  perfonnance  measures  and  provide  the  decision  maker 
with  another  metric  from  which  to  make  decisions. 

The  breadth  of  metrics  should  make  measuring  perfonnance  of  an  information 
system  relatively  simple,  unfortunately  that  is  not  the  case  and  there  remain  several 
challenges.  Although  capabilities  are  identified,  the  perfonnance  measures  establishing 
capability  achievement  have  not  been  fully  identified  or  aligned  for  all  capabilities  (DoD 
CIO,  2012).  As  such,  accurate  and  meaningful  cost/benefit  ratios  are  difficult  to 
implement  (Olson  &  Wu,  2011).  Goal  value  computation  is  still  a  work  in  progress  with  a 
needed  shift  in  focus  from  objective  function  optimization  to  realistic  target  values  and 
goals  for  those  functions  (Zandi  &  Tavana,  2011).  The  goal  appears  to  have  a  single 
measure  from  which  to  make  decisions,  but  no  manager  should  trust  a  single  measure  in 
its  entirety,  rather  take  a  look  at  multiple  measures  to  detennine  efficiency  and 
effectiveness  (Merikas  et  ah,  2005).  Maintaining  flexibility  is  crucial  with  new 
infonnation,  policy,  technology,  or  other  environment  changes  that  could  signal  a  change 
in  a  system  dynamic  relationship.  Measurements  and  forecasts  respond  to  these  insights 
in  order  to  provide  the  decision  makers  with  the  needed  infonnation  (Scher  &  Koomey, 
2011).  Volatility  measurements  can  provide  the  insight  needed  to  monitor  multiple 
metrics,  with  changes  in  volatility  signaling  the  change  of  a  relationship  and  thereby 
focusing  the  decision  maker. 

E.  VOLATILITY 

Measuring  and  forecasting  volatility  through  historical  volatility  and  a  normal 
distribution  provides  a  volatility  expectation  from  which  managers  can  make  decisions 
(Ederington  &  Guan,  2006).  Volatility  measurements  are  used  to  represent  future 
volatility  and  are  used  throughout  the  financial  services  industry  to  forecast  stock  index 
volatility  (Arak  &  Mijid,  2006).  Those  decisions  in  conjunction  with  a  thorough  risk 
assessment  should  reduce  the  complexity  and  uncertainty  in  the  implementation  of  an 
infonnation  system  (Taylor,  Artman,  &  Woelfer,  2012).  The  volatility  measurements  can 
provide  options  from  which  to  reduce  uncertainty  by  reassessing  the  business  metrics  as 
infonnation  changes  or  becomes  available  (Zandi  &  Tavana,  2011).  The  reassessment 
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should  include  quantifiable  measures  that  hedge  the  risk  of  the  system  and  reduce  the 
overall  volatility  of  the  investment  portfolio  (Damodaran,  2005).  In  this  section,  volatility 
is  reviewed  with  its  uses  in  the  financial  services,  how  it  is  measured  and  forecasted,  and 
its  relationship  with  risk. 

Within  the  financial  services  sector,  volatility  is  measured  in  the  S&P  500  through 
the  VIX.  The  volatility  is  estimated  through  the  standard  deviation  of  the  closing  price  of 
an  asset  (Ganesan  &  Yadav,  2007).  The  VIX  provides  investors  a  forecast  on  the 
expected  future  volatility  of  the  stock  market  with  future  contracts  that  give  a  30-day 
expected  volatility  (D’Anne,  2012).  The  VIX  is  also  seen  as  an  indicator  for  sentiment  on 
the  direction  of  stock  prices,  especially  declines  (Arak  &  Mijid,  2006). 

Measuring  volatility  provides  not  only  sentiment  of  market  decline  in  its 
associated  market,  but  also  the  actual  imminent  volatility  (Arak  &  Mijid,  2006).  Defining 
volatility  is  simple  as  equating  it  to  the  standard  deviation  of  the  measured  return  over  a 
specified  period  of  time  (Wang,  Wang,  &  Yourougou,  2012).  This  is  in  alignment  with 
the  treatment  of  uncertainty  within  probability  theory  (Zandi  &  Tavana,  2011). 
Measuring  the  volatility  can  then  provide  the  value  of  options  as  the  square  root  of  the 
forward  volatility  (D’Anne,  2012).  There  are  other  approaches  to  measuring  volatility 
such  as  the  mean  absolute  return  deviation  which  has  been  argued  as  more  accurate 
alternative  to  historical  volatility  (Ederington  &  Guan,  2006).  Other  alternatives  include 
the  daily  squared  returns,  serial  correlation  adjusted,  mean  adjusted  daily  squared  returns, 
and  absolute  change  in  returns  (Iltuzer  &  Tas,  2013). 

The  general  consensus  is  the  use  of  historical  standard  deviation  to  determine  the 
volatility  (Ederington  &  Guan,  2006).  There  are  arguments  against  using  historical  data 
on  the  premise  that  the  past  in  not  a  quality  predictor  of  the  future  (Hosley,  2011).  The 
other  concern  in  measuring  volatility  is  the  subjective  choice  of  the  number  of 
observations  to  determine  the  standard  deviation  sample  size,  the  forecast  horizon,  and 
the  moving  average  (Iltuzer  &  Tas,  2013).  However,  measuring  volatility  is  a  worthwhile 
endeavor  as  it  allows  forecasting  and  the  comparison  of  those  forecasts.  Information 
systems  are  generally  found  to  increase  efficiency  and  reduce  volatility  (Long,  2010), 
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measuring  the  volatility  of  the  appropriate  metrics  would  allow  for  a  visual  monitoring  of 
the  efficiency  and  volatility  for  a  given  IT  system. 

Forecasting  volatility  allows  for  the  forecasting  of  other  metrics  and  provides  a 
mechanism  by  which  to  compare  and  evaluate  results  of  a  system  (Iltuzer  &  Tas,  2013). 
The  comparison  and  evaluation  of  results  are  especially  relevant  in  a  volatile  environment 
and  those  periods  of  volatility  are  expected  to  continue  (Elias,  2000).  Volatility 
measurements,  both  the  actual  and  forecasted  volatility,  provide  infonnation  on  the  future 
volatility  of  a  system  (Wang  et  ah,  2012).  Although  there  are  several  forecasting 
methodology  and  volatility  models,  the  best  volatility  forecast  includes  the  current 
volatility  (Iltuzer  &  Tas,  2013). 

Future  volatility  is  forecasted  across  a  wide  variety  of  forecast  horizons  using 
historical  standard  deviation  (Ederington  &  Guan,  2006).  For  example,  the  VIX  has  been 
shown  to  have  a  short  forecast  horizon  with  most  viewing  it  as  a  forecast  of  imminent 
volatility  (Arak  &  Mijid,  2006).  Other  volatility  measurements  use  other  metrics  such  as 
the  information  ratio  which  measures  the  volatility  in  excess  return  per  unit  (Merikas  et 
ah,  2005).  The  relationship  between  volatility  and  returns  appears  to  be  skewed  to  the 
negative  side  with  negative  returns  having  a  greater  impact  on  the  increase  of  volatility  as 
compared  to  the  decrease  in  volatility  with  a  positive  return  (Lee  &  Rye,  2013).  Given 
that  relationship,  a  forecast  of  increased  volatility  is  a  good  option  point  to  ensure  that  the 
value  of  a  system  increases  (D’Anne,  2012). 

Developing  a  forecast  for  future  volatility  provides  a  tool  by  which  to  compare 

actual  volatility  against  a  forecast  which  can  then  help  detennine  if  there  has  been  an 

increase  or  decrease  in  risk  (Arak  &  Mijid,  2006).  It  appears  that  there  is  a  positive 

relationship  between  results  and  volatility  with  high  beta  portfolios  performing  extremely 

well  in  low  volatility  environments;  however  a  high  volatility  environment  is  detrimental 

to  those  returns  (Trainor,  2012).  This  would  indicate  that  high  volatility  environments  are 

riskier  and  decision-makers  should  act  appropriately.  New  infonnation  systems  that  are 

planned  tend  to  resemble  longer  established  systems  in  their  results,  however  emergent 

systems  seem  to  be  the  product  of  high  volatility  environments  and  are  at  the  risk  of 

unbalancing  the  benefits  of  stability  with  the  disruption  of  volatility  especially  in  its 
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initial  use  (Kang,  2007).  Knowing  the  different  volatility  fluctuation  points  would 
provide  the  decision-maker  with  the  capability  to  hedge  risk  and  provide  for  greater 
returns  to  the  organization  (Damodaran,  2005). 

Volatility  is  affected  by  several  different  factors,  normally  in  the  form  of  shocks 
through  changes  in  capital,  labor,  preferences,  demand,  or  technologies  (Chichilnisky  & 
Gorbachev,  2004).  The  uncertainty  that  these  factors  produce  provides  an  opportunity  for 
exploitation  of  those  factors  through  risk  management  (Damodaran,  2005).  The  shocks  to 
the  system  generally  increase  volatility;  however  negative  shocks  have  demonstrated 
stronger  increases  in  volatility  (Lee  &  Rye,  2013).  This  provides  decision-makers  an 
opportunity  to  identify  the  possible  negative  shocks  and  place  options  on  those  factors  in 
order  to  take  advantage  of  the  uncertainty.  However,  once  in  a  high  volatility 
environment  the  return  on  the  systems  does  not  correlate  to  a  respective  beta  level,  which 
means  that  a  high  beta  portfolio  would  have  higher  risk  without  a  corresponding 
likelihood  of  higher  returns  (Trainor,  2012).  Therefore,  the  hedging  of  risks,  through 
options,  is  crucial  to  protect  against  negative  risks  while  the  environment  is  of  low 
volatility  resulting  in  increasing  value  to  the  options  as  volatility  increases  (Damodaran, 
2005). 

F.  RISK 

IT  investments  are  inherently  risky  especially  in  a  volatile  environment  (Zandi  & 
Tavana,  2011).  Measuring  risk  is  seen  as  the  product  of  two  factors,  the  intensity  of  the 
disruption  to  the  system  and  the  likelihood  of  the  risk  to  happen  (A1  Kattan  et  ah,  2011). 
In  financial  analysis,  risk  is  viewed  as  the  nondiversifiable  or  systematic  risk  of  a 
particular  market  with  its  value  affected  by  the  discount  rate.  Similarly  to  the  infonnation 
system  environment,  risk  management  is  the  primary  defense  against  risk  with  risk¬ 
hedging  options  available  (Damodaran,  2005).  The  high  cost  of  many  infonnation 
systems  is  a  major  risk  in  of  itself  (Olson  &  Wu,  2011).  Volatility  measurements  and 
forecasting  is  crucial  in  order  to  conduct  proper  risk  management  (Brownlees,  Engle,  & 
Kelly,  2012).  This  section  reviews  the  risk  factors,  operational  risk,  challenges  of  risk, 
measuring  risk,  and  the  relationship  of  risk  to  beta  as  they  pertain  to  information  systems. 
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Many  organizations  appear  to  have  a  low  awareness  of  risk  factors  and  risk 
management  activities,  especially  as  it  relates  to  IT  critical  risk  factors  (Gholami,  2012). 
The  DoD  uses  portfolio  risk,  a  combination  of  utility  and  cost,  to  review  infonnation 
systems  and  ensure  that  they  meet  the  capabilities  required  and  assessing  the  costs  in 
terms  of  cost  savings  and  improvements  to  the  process.  The  process  of  identifying  the 
portfolio  risk  encompasses  the  entire  life  cycle  for  the  system  and  provides  for  an 
estimated  ROI  (DCMO,  2013).  There  are  many  risk  factors,  including  outsource  risks  and 
changes  to  requirements,  and  conducting  an  analysis  on  those  factors  can  quantify  the 
risk  and  provide  the  managers  the  information  needed  to  make  decisions  (Olson  &  Wu, 
2011). 

The  evaluation  of  risk  is  quantified  through  an  assessment  of  the  value  of  the 
asset,  the  vulnerability  of  that  asset  and  the  threat  to  that  asset  (Gholami,  2012).  The 
threat  to  an  asset  is  identified  in  risk  management  of  IT  by  assessing  many  factors 
including  the  budget,  staff,  expectations,  planning,  requirements,  management  support 
and  user  involvement  (A1  Kattan  et  ah,  2011).  Other  risk  factors  include  physical  threats, 
such  as  fires  or  break-ins,  system  intrusions  which  would  be  hackers  or  malicious 
software,  and  functional  threats  that  includes  improper  use  or  inaccurate  data  (Olson  & 
Wu,  2011).  The  risks  are  categorized  into  financial  risks,  technical  risks,  managerial 
risks,  behavior  risks,  and  political  and  legal  risks  (Gholami,  2012).  The  political  and 
financial  risks  are  especially  important  as  they  pertain  to  the  DoD  environment.  On  the 
governance  side,  the  most  important  factors  in  implementing  an  infonnation  system  are 
clear  requirements  and  a  competent  staff  (A1  Kattan  et  ah,  2011).  Due  diligence  in 
identifying  the  risk  factors  and  employing  a  risk  management  plan  should  reduce 
volatility.  The  volatility  measurements  assist  in  tracking  the  identified  risk  factors  and 
determining  if  the  risk  management  plans  are  effective. 

Organizations  invest  in  information  systems  to  increase  their  value  by  becoming 
more  efficient  at  generating  cash  flow  and/or  by  lowering  the  cost  to  do  business. 
Effective  risk  management  assists  in  those  endeavors  through  identification  of  the  factors 
that  hinder  the  value  increase  (Damodaran,  2005).  One  definition  of  risk  is  the  probability 
of  failing  to  meet  an  objective  (DCMO,  2013).  Increasing  value  is  such  an  objective  and 
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identifying  risk  factors  and  putting  in  place  mitigation  plans  to  combat  factors,  such  as 
loss  of  operational  data  and  downtime,  is  critical  to  ensure  that  objectives  are  met  (Olson 
&  Wu,  2011).  Measuring  risks  quantitatively  continues  to  be  a  challenge  as  is  the 
measurement  of  IT  perfonnance  as  it  relates  to  increasing  value  (DoD  CIO,  2012). 
Measuring  the  value  in  the  decrease  of  operational  risk  is  a  step  in  the  right  direction. 

Risk  assessment  techniques  have  grown  in  importance,  the  ability  to  identify  and 
avoid  unnecessary  risk  is  valuable  (Gilliam  et  ah,  2010).  The  IT  budget  is  such  a  risk  that 
continues  to  be  one  of  the  most  significant  factors  to  the  success  of  an  information  system 
(A1  Kattan  et  ah,  2011).  Having  identified  the  risk,  the  budget  can  be  increased  to  avoid 
the  risk  altogether  or  other  mitigation  steps  can  be  taken  to  control  costs  and  reduce  the 
risk.  Unfortunately,  in  a  high  technological  environment,  it  is  difficult  to  identify  the 
exact  source  of  a  risk  such  as  escalating  costs  (Long,  2010).  However,  one  operational 
risk  factor  that  can  be  mitigated  directly  and  reduce  other  risks  indirectly  is  to  ensure  a 
competent  staff  is  in  place.  A  competent  staff  is  more  likely  to  implement  an  infonnation 
system  successfully  as  well  as  conduct  proper  risk  management  to  reduce  the  overall 
operational  risk  of  a  system  (A1  Kattan  et  ah,  2011). 

There  are  several  risk  challenges  especially  as  it  relates  to  infonnation  systems. 
Even  with  all  of  the  different  risk  assessment  techniques,  methods,  and  tools  available  to 
identify  and  measure  risk,  there  remains  the  challenge  of  low  awareness  for  risk 
management  (Gholami,  2012).  Those  organizations  that  practice  good  risk  management 
are  still  faced  with  the  challenge  that  not  all  risk  can  be  eliminated.  Since  not  all  risk  can 
be  eliminated,  decision-makers  prioritize  the  IT  strategies  with  risk  considerations  or  at 
least  should  (Zandi  &  Tavana,  2011).  Long-term  projects  have  the  advantage  of  time  but 
the  decision-makers  for  those  projects  need  to  be  wary  that  they  do  not  take  extreme  risks 
and  counter  the  time  advantage  (Elias,  2000).  Possibly  the  biggest  risk  challenge  is  to 
have  the  entire  organization,  not  only  the  decision-makers,  to  look  for  the  opportunities 
that  arise  from  proper  risk  management  (Damodaran,  2005). 

Risk  has  both  a  danger  and  an  opportunity  element  (Damodaran,  2005).  Risk 

management  is  a  positive  process  with  a  feedback  element  to  generate  control  leverage 

and  get  realistic  expectations  (A1  Kattan  et  ah,  2011).  The  asymmetric  relationship  of  an 
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asset  value  and  volatility  is  displayed  through  leverage  hypothesis  and  the  risk  factor 
volatilities  display  a  similar  relationship  with  control  leverage  (Lee  &  Rye,  2013).  In 
portfolios,  decision  making  models  quantify  risk  through  probability  and  utility  (Taylor 
et.  ah,  2012).  Once  the  risk  is  quantified,  it  can  then  be  correlated  to  the  overall  portfolio 
which  allows  for  the  opportunity  to  include  negatively  correlated  assets  to  reduce  the 
overall  risk  of  the  portfolio  (D’Anne,  2012). 

Risk  is  the  uncertainty  and  unexpected  results  in  an  infonnation  system  which  is 
displayed  in  volatility.  Volatility  measurements  of  the  metrics  provide  a  method  to 
quantify  that  risk.  That  quantification  provides  the  decision  makers  the  information 
needed  to  either  hedge  that  risk  in  order  to  protect  against  it  or  to  place  an  option  on  that 
risk  in  order  to  take  advantage  of  it  (Damodaran,  2005).  Either  way,  whether  to  hedge  or 
option  the  risk,  regular  volatility  measurements  reviews  and  evaluations  ensure  proper 
governance  of  the  system  (Gilliam  et  ah,  2010).  A  risk  monitoring  element  integration 
into  the  information  system  provides  the  progress  feedback,  analysis,  and  corrective 
action  points  (Taylor  et.  ah,  2012).  The  volatility  of  the  metrics  to  be  measured  and 
monitored  is  identified  through  evaluation  of  the  criteria  and  objectives  of  the 
information  system  (Zandi  &  Tavana,  2011).  The  measurement  of  risk  provides  the 
mechanism  to  generate  risk  profiles  for  the  infonnation  system  which  leads  to  a  decision 
on  the  proper  level  of  governance  and  controls  for  the  system  (Taylor  et.  ah,  2012).  In  a 
portfolio  of  investments,  the  volatility  feedback  takes  advantage  of  the  positive 
relationship  between  the  volatility  and  the  expected  return  (Lee  &  Rye,  2013).  In  the 
financial  sector,  the  VIX  provides  the  volatility  measurements  and  when  measuring  a 
high  level  signals  significant  risk  of  sharp  market  moves  (D’Anne,  2012). 

IT  investment  managed  as  portfolios  along  with  incorporation  of  perfonnance 
measurements  and  risk  management  was  mandated  by  DoD  in  2005  (DoD,  2005).  With  a 
portfolio  of  investments,  the  beta  of  the  investment  can  represent  the  risk.  Beta  is  the 
ratio  of  the  individual  system  covariance  and  the  overall  portfolios  returns  (Roztocki  & 
Weistroffer,  2009).  Once  beta  is  detennined,  investment  portfolio  management  can 
leverage  the  relationship  between  perfonnance  and  the  volatility  environment  where  high 
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beta  portfolios  perform  better  in  low  volatility  environments  vice  high  volatility  (Trainor, 
2012).  Determining  the  volatility  environment  then  becomes  the  challenge. 


G.  BETA 

In  the  financial  sector,  beta  measures  the  systematic  risk  of  an  asset  or  portfolio 
and  is  measured  by  the  volatility  of  that  asset  or  portfolio  (Ganesan  &  Yadav,  2007).  Beta 
is  a  good  indicator  of  deviation  in  a  portfolio,  accounting  for  70%  of  the  correlation 
(Trainor,  2012).  This  section  reviews  beta  in  the  financial  markets  and  the  measurement 
of  beta  and  its  relation  to  volatility. 

The  Capital  Asset  Pricing  Model  (CAPM)  describes  the  risk  and  return 
relationship  between  an  asset  and  its  market,  with  beta  representing  the  risk  (Damodaran, 
2005).  The  volatility  environment  has  an  effect  on  the  returns  with  high  volatility 
environments  producing  negative  returns  for  high  beta  portfolios.  The  time  horizon  also 
has  an  effect  with  short  tenn  investments  having  a  higher  risk  factor  for  beta  than  long 
term  investments  (Trainor,  2012).  With  a  changing  environment,  the  beta  will  also 
change  through  time  (Siegel,  1995).  Compounding  of  returns  only  amplifies  the  negative 
relationship  between  higher  beta  assets  and  high  volatility  environments  as  well  as  the 
positive  relationship  of  higher  beta  assets  in  low  volatility  environments. 

Regression  analysis  of  historical  data  is  presently  the  best  method  to  measure  beta 
(Siegel,  1995).  The  regression  analysis  determines  if  the  beta  is  related  to  the  any  excess 
return  in  the  present  time  period  based  on  the  returns  from  prior  time  periods  for  a 
specific  asset.  The  volatility  over  the  specified  time  period  is  a  beta  multiple  of  the 
portfolio  (Trainor,  2012). 

Beta  and  return  relationships  turn  negative  with  substantial  increases  in  volatility 
levels  (Trainor,  2012).  Increasing  returns  to  scale  industries,  those  that  industries  where 
efficiency  increases  with  greater  production,  are  generally  more  volatile  (Chichilnisky  & 
Gorbachev,  2004).  Low  beta  assets  in  low  volatility  environments  outperform  high  beta, 
high  volatility  assets.  Furthennore,  asset  returns  decrease  as  volatility  increases  (Trainor, 
2012).  Therefore,  forecasting  volatility  allows  the  decision  maker  to  adjust  the  portfolios 
in  accordance  with  the  prevailing  volatility  environment. 
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H.  FORECAST 


In  a  2011  survey,  all  companies  planned  to  invest  in  information  systems, 
especially  process  and  data  analytics,  with  the  hopes  to  improve  forecasting  and 
budgeting  (Hosley,  2011).  The  forecasting  of  volatility  is  especially  important  in  high 
volatility  environments,  which  is  seen  in  many  asset  classes  (Brownlees  et.  ah,  2012). 
This  sections  reviews  the  data  for  forecasting,  forecasting  of  value,  and  the  models  used 
to  forecast  and  their  relationship  with  volatility. 

Volatility  measurements  forecast  future  volatility  of  the  underlying  asset  (Arak  & 
Mijid,  2006).  The  ability  to  capture  that  future  volatility  and  the  change  in  systematic  risk 
gives  the  decision  maker  more  agility  in  governance  of  that  portfolio  (Siegel,  1995). 
However,  there  are  requirements  for  the  system  in  order  to  benefit  from  the 
measurements.  Those  requirements  include  being  observable,  constancy  across 
variations,  constancy  of  structure,  and  collection  of  accurate  and  large  amounts  of  data 
(Scher  &  Koomey,  2011).  There  should  also  be  a  risk  monitoring  mechanism  integrated 
into  the  system  that  allows  for  progress  feedback,  analysis,  and  corrective  action  (Taylor 
et.  ah,  2012).  The  measurements  and  risk  monitoring  are  part  of  the  overall  risk 
management  plan  which  establishes  the  policies  and  response  to  uncertain  events 
(A1  Kattan  et  ah,  2011). 

In  a  high  volatility  environment,  high  beta  portfolio  values  decline  (Trainor, 
2012),  which  makes  knowing  when  a  portfolio  has  entered  a  high  volatility  environment 
critical  to  the  management  of  a  portfolio.  Uncertainty  is  part  of  any  investment  and 
knowing  how  to  respond  to  shocks  in  the  system  is  good  governance.  A  positive  shock 
generally  results  in  a  volatility  decrease,  whereas  a  negative  shock  results  in  a  much 
stronger  increase  in  volatility  (Lee  &  Rye,  2013).  Knowing  the  volatility  environment  is 
important  in  that  a  high  beta  portfolio  performs  well  in  a  low  volatility  environment  yet 
underperforms  greatly  in  high  volatility  environments  (Trainor,  2012). 

Rolling  forecasts  generate  more  accurate  results  in  the  value  of  an  asset  (Hosley, 
2011).  The  value  of  an  asset  is  the  value  of  key  inputs  such  as  cash  flows  already  made, 
growth  rate,  time,  and  discount  rate  that  reflects  the  risk  for  that  asset.  The  discounted 
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cash  flow  (DCF)  model  measure  the  value  of  a  firm  in  such  a  way.  However,  risk 
adjustments  to  that  valuation  model  narrowly  focuses  on  discount  rate  which  results  in 
adjustments  that  are  either  nonexistent  or  haphazard  (Damodaran,  2005).  Using  large 
forecast  time  periods  along  with  frequent  reestimation  of  those  forecasts  improves  the 
accuracy  of  value  forecasts  and  reduces  the  need  for  risk  adjustments  (Brownlees  et.  ah, 
2012).  The  DCF  model  uses  expected  value  as  its  metric  of  choice  to  represent  the 
probabilities  of  multiple  forecasts  to  revenues,  future  cash  flows,  and  growth  and  margins 
(Damodaran,  2005).  However,  in  knowledge  sectors,  which  information  systems  would 
belong,  growing  productivity  could  result  in  more  volatility  resulting  in  the  need  for  more 
forecasts  (Chichilnisky  &  Gorbachev,  2004).  A  similarity  among  organizations  is  the 
desire  to  have  an  EPM  program  that  provides  visibility  to  the  forecasts  over  time  (Hosley, 
2011). 

The  accurate  forecasting  of  volatility  with  a  given  asset  is  of  growing  importance 
(Ganesan  &  Yadav,  2007).  In  the  financial  management  sector,  this  importance  is  seen 
with  the  VIX  and  its  volatility  forecast  model  (D’Anne,  2012).  The  simple  premise  in 
financial  managements  is  that  over  time  greater  risk  should  come  with  greater  returns 
(Trainor,  2012).  The  goal  is  to  identify  the  factors,  time  horizons,  and  reestimation 
frequency  to  develop  a  successful  forecasting  and  risk  assessment  model  (Brownlees  et. 
ah,  2012).  Unfortunately,  forecasting  is  hindered  due  to  the  lack  of  structural  constancy 
and  constancy  across  different  conditions  as  systems  are  constantly  changing  (Scher  & 
Koomey,  2011).  However,  identifying  relationships  such  as  continued  low  volatility 
levels  with  high  beta  portfolios  results  in  higher  returns  (Trainor,  2012)  and  result  shocks 
increase  volatility  in  both  directions  with  negative  shocks  producing  the  greatest  change 
in  volatility  are  valuable  (Lee  &  Rye,  2013).  These  relationships  provide  the  decision¬ 
maker  with  an  opportunity  to  exploit  these  relations  and  adjust  portfolios  based  on  the 
volatility  measurements  (Trainor,  2012).  Capturing  and  understanding  the  volatility 
changes  provides  the  foundation  for  better  models  and  forecasts  (Brownlees  et.  ah,  2012). 

There  are  many  factors  that  affect  the  success  of  an  infonnation  system,  its 
governance,  integration  with  other  systems  and/or  sensors,  the  metrics  used  to  gauge 
perfonnance,  and  the  risk  in  the  environment  and  with  the  system.  Volatility 
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measurements  appear  to  encompass  all  of  those  factors  and  provide  relationships  and 
correlations  that  provide  an  opportunity  for  exploitation.  In  the  financial  sector  this  is 
seen  with  the  trading  of  VIX  contracts  that  are  of  negative  correlation  with  the  market 
which  reduces  portfolio  risk  significantly  (D’Anne,  2012).  Applying  volatility 
measurements  to  information  systems  seems  to  provide  the  potential  for  exceptional 
returns. 
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III.  METHODOLOGY 


How  quickly  a  system  is  adopted  will  be  affected  by  the  fluidness  and 
completeness  of  the  implementation  plan.  An  IT  system  is  a  strategic  tool  to  accomplish 
an  organization’s  mission  and  should  link  to  the  overall  strategic  plan  as  well  as  the 
budget  process  (Yang  &  Melitski,  2007).  If  the  system  does  not  meet  that  basic 
requirement  of  being  linked  to  a  higher  strategy  then  the  implementation  is  likely 
doomed.  The  ability  to  review  the  performance  of  major  IT  investments  and  strengthen 
IT  investments  is  critical  in  accomplishing  the  overall  IT  strategy  (Tekai,  2012).  The  data 
collection  should  be  integrated  and  streamlined,  focusing  on  benefits,  including  the 
intangibles,  and  capable  of  allowing  the  comparison  of  alternatives  and  the  development 
of  options.  Pilot  programs  should  be  implemented  to  get  credible  numbers  especially  if  a 
previous  system  is  not  available.  The  IT  systems  should  have  a  clearly  designated  person 
responsible  for  the  tracking  and  perfonnance  of  those  IT  systems  along  with  the 
responsibility  for  exercising  options  in  order  to  make  more  sound  decisions  (Copeland  & 
Tufano,  2004).  The  metrics  being  collected  need  to  be  relevant  to  the  objectives  desiring 
change  including  any  cultural  change  that  is  desired  which  will  require  an  organizational 
measure  of  performance  (Kasunic,  2002).  Metrics  provide  the  input  from  which 
management  can  review  performance.  Understanding  of  the  technology  life  cycle 
provides  managers  with  the  knowledge  to  strengthen  IT  investments.  The  volatility 
measurement  is  reliant  upon  data  collection  and  the  implementation  of  the  IT  system  and 
enhances  the  metrics  and  technology  life  cycle  of  the  IT  system. 

A.  DATA  COLLECTION 

The  data  collection  should  allow  the  IT  system  to  deliver  business  value  and 
becomes  a  source  of  innovation  (Stikeleather,  2013).  An  important  feature  of  data 
collection  should  be  that  it  is  operational  in  nature,  while  still  being  linked  to  strategic 
objectives  (Yang  &  Melitski,  2007).  Calculating  true  cost  and  benefits  of  IT  is  a 
challenge  for  both  revenue  and  cost  elements  and  an  integrated  data  collection  process 
would  ease  that  burden.  Data  collection  should  be  credible  with  clear,  concise, 
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understandable,  and  believable  metrics  without  being  a  burden  to  the  operator.  The  data 
becomes  of  central  importance  to  the  IT  manager  as  data  collection,  data  reduction,  data 
analysis,  and  data  display  is  managed  by  the  IT  system  allowing  the  manager  to  make 
decisions  quickly  (Yang  &  Melitski,  2007).  The  data  collected  can  then  be  measured  for 
volatility,  adding  another  layer  of  infonnation  for  the  IT  manager  to  make  strategic 
decisions  as  it  relates  to  the  IT  system  and  the  options  presented  for  the  system. 

Data  Gathering  Procedures: 

The  database  provides  an  approach  from  which  a  rigorous  cost/benefit  study  is 
implemented  to  evaluate  the  information  system  (Olson  &  Wu,  2011).  The  SIGINT 
database  comprises  5  signals  intelligence  systems  that  store  reports.  The  number  of 
reports  provides  the  inputs  for  a  surrogate  revenue  over  a  specified  time  frame.  Computer 
networks  are  becoming  more  predominant  in  the  perfonnance  of  an  information  system 
along  with  the  integration  of  those  systems.  The  complexity  of  the  system  is  likely  to 
produce  issues  during  actual  use  (Kang,  2007).  Because  of  this  complexity  the  generated 
number  of  reports  is  scrubbed  to  ensure  that  the  analysis  does  not  include  time  frames  of 
what  appears  to  be  initial  testing.  The  data  gathering  provides  over  two  years  of  data  from 
which  to  conduct  analysis.  The  actual  names  of  the  systems  as  well  as  the  dates  have 
been  substituted  in  order  to  maintain  the  research  as  unclassified.  The  time  periods  do 
align,  i.e.,  n  (period)  102  of  system  1  =  n  102  of  system  3.  Systems  2,  3,  and  5  contain 
incremental  systems  and  are  annotated  as  2.0,  2.1,  etc.  The  data  is  collated  onto  a 
spreadsheet  from  which  the  analysis  is  conducted. 

B.  ANALYSIS 

The  purpose  of  metrics  is  to  compare  other  systems  to  one  another,  however  with 
many  systems  lacking  a  common  output  or  environment,  metrics  by  themselves  can  be 
misleading.  It  is  the  relativity  of  the  metrics  of  one  system  to  the  same  metric  in  another 
system,  such  as  a  previous  system,  where  the  benefits  can  be  measured.  A  previous 
system  or  historic  data  can  provide  a  baseline  from  which  a  new  system  can  be  compared. 
It  is  critical  that  the  metric  chosen  effects  the  objective  desired  and  is  consistent  in  its 
measurement.  There  are  many  metrics  that  can  be  chosen  such  as  ROI,  efficiency  ratios, 
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cost  constraints,  KVA,  payback  period,  unit  cost,  etc.  In  an  IT  system,  the  benefits  are 
rarely  revenue  and  therefore  a  surrogate  is  used,  which  only  adds  to  the  measurement 
ambiguity.  Using  metrics  over  a  short  time  period  could  lead  to  a  lot  of  volatility  and 
those  short-term  fluctuations  could  lead  to  an  overreaction  by  managers.  Using  a  longer 
timeframe  and  cyclically  adjusting  the  metric  over  that  timeframe  should  give  a  better 
measure  of  the  true  volatility  and  value  of  the  system.  However,  metrics  themselves 
could  be  shortcomings  as  there  are  other  factors  such  as  interoperability  and  agility  that 
may  not  directly  influence  cost  or  benefits  and  are  even  more  difficult  to  measure  in 
nominal  terms.  Volatility  measurements  over  a  long  period,  with  any  appropriate  metric, 
provide  additional  information  for  the  manager  from  which  to  review  perfonnance  and 
strengthen  the  IT  investments. 

Rios  conducted  research  on  SIGINT  systems  to  detennine  its  ROI.  Analyzing  the 
outputs  of  the  U.S.  Navy  CCOP  in  common  units,  a  price  per  unit  of  output  can  be 
generated  to  account  for  both  cost  and  revenue  at  the  subprocess  as  well  as  asset  level. 
This  can  then  be  used  to  detennine  ROI  for  each  asset  and  allow  for  effective  valuation 
and  comparison  of  different  IT  systems.  Further,  this  can  provide  for  a  common 
framework  from  which  to  understand,  evaluate,  and  justify  the  impact  of  government 
investments  in  IT  systems.  This  common  framework  is  the  KVA  methodology  which 
allows  program  managers  to  build  meaningful  metrics  and  perfonn  financial  analysis  on 
the  SIGINT  systems  (Rios,  2005). 

Lambeth  and  Clapp  built  upon  Rios’  research  and  applied  the  KVA  methodology 
to  the  deployment  of  the  CCOP  system  on  board  the  USS  GONZALEZ  (DDG-66). 
Military  acquisition  requires  investments  to  be  productive,  efficient,  and  support  joint 
operational  capabilities.  Defining  a  ROI  methodology  by  which  systems  can  be  measured 
is  crucial  in  establishing  a  more  efficient  acquisition  process.  Furthermore,  using  a  near 
real-time  method  allows  the  decision-maker  to  get  the  latest  metrics  on  a  system  to 
detennine  its  present  effectiveness  and  provide  valuable  infonnation  for  a  more  effective 
revenue  allocation.  The  KVA  methodology  would  allow  program  managers  to  allocate 
future  resources  based  on  the  collected  metrics  and  therefore  make  more  efficient  IT 
investment  decisions  (Lambeth,  III  &  Clapp,  2007). 
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Marco  Nelson  captured  volatility  of  ROI  to  generate  a  beta  derivation  in  IT 
investment  portfolios.  DoD  CIO  instructs  that  “a  portfolio  baseline  shall  be  established 
and  maintained  for  each  portfolio”  (Instruction,  2006,  p.  15).  Since  the  DoD  does  not 
generate  revenue,  similar  to  the  corporate  market,  a  surrogate  for  revenue  and  cost 
streams  such  as  the  KVA  methodology  should  be  used.  The  KVA  methodology  can  also 
be  used  to  provide  the  ROI  estimates  of  volatility  in  order  to  produce  the  notional  IT  beta. 
Key  decision  makers  can  then  use  the  beta  derivation  steps  to  develop  the  baseline  of  a 
family  of  systems  and  then  evaluate  new  or  existing  assets  within  an  investment  portfolio 
(Nelson,  2010). 

This  research  expands  on  the  previous  work  adding  volatility  measurements  as 
well  as  analyzing  actual  returns  that  the  SIGINT  database  has  gathered.  Unit  cost  will  be 
used  as  the  surrogate  for  revenue  as  it  can  quickly  be  calculated  for  each  system  with  the 
data  in  the  database. 

Analysis  Procedures: 

The  number  of  reports  is  compared  on  a  daily,  weekly,  and  monthly  time  frame 
against  the  rolling  average,  the  average  number  of  reports  over  the  life  of  the  system,  and 
the  moving  average,  the  average  number  of  reports  over  the  last  60-days  (8  weeks  and 
2  months).  This  gives  a  visual  representation  of  the  benefits  of  the  system  over  its  life 
cycle. 

The  next  step  is  to  calculate  the  unit  cost.  Based  on  annual  budget  estimations,  the 
systems  cost  $2,000,000  annually  to  operate  (DoD,  2013).  The  exact  amount  for  each 
system  is  classified  therefore  an  estimation  is  used.  The  amount  along  with  the  number  of 
reports  is  used  to  calculate  a  unit  cost  ($/report).  The  unit  cost  is  used  as  the  surrogate  for 
revenue  as  well  as  the  volatility  data. 

One  challenge  is  to  “smooth”  the  data  as  it  goes  through  the  business  cycle 
(Chichilnisky  &  Gorbachev,  2004).  In  order  to  smooth  the  calculated  data,  the  number  of 
reports  is  based  on  a  moving  time-frame  of  one  year.  The  initial  unit  cost  is  detennined 
by  the  first  365  data  points  gathered  from  the  database.  The  unit  cost  metric  becomes  the 
input  into  the  volatility  measurements. 
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The  volatility  measurements  are  determined  from  the  unit  cost  of  the  respective 
time  frames.  First,  the  natural  log  of  the  change  between  the  unit  costs  is  detennined. 
Then  the  standard  deviation  of  those  natural  logs  is  determined  to  provide  the  volatility  of 
the  system. 

Statistical  analysis  provides  the  relationship  between  volatility  and  benefits  over 
time  (Scher  &  Koomey,  2011).  The  historical  standard  deviation  provides  the  volatility 
measurements  that  are  then  used  to  forecast  the  future  volatility.  Future  volatility  is 
forecasted  through  historical  standard  deviation  (Ederington  &  Guan,  2006).  This  is  a 
time  series  approach  for  forecasting  volatility,  which  relies  on  past  volatility  to  predict 
future  results  (Wang  et  al.,  2012).  The  model  used  to  detennine  the  forecast  is  the 
Generalized  Autoregressive  Conditional  Heteroskedasticity  (GARCH)  model.  The 
forecasts  generated  in  this  research  are  for  a  year  out  in  order  to  provide  decision  makers 
an  opportunity  to  plan  and  take  actions  based  on  the  data.  The  forecasts  are  estimated 
using  the  data  from  the  daily,  weekly,  and  monthly  computations  providing  a  range  of 
forecasts  from  which  to  make  decisions. 

The  final  analysis  is  to  detennine  beta  between  all  of  the  systems.  The  percentage 
of  change  for  the  unit  cost  is  detennined  for  each  system.  The  market  comprises  the  unit 
cost  calculated  using  all  reports  generated  and  the  total  cost  of  all  of  the  systems  added 
together.  The  change  percentage  is  calculated  for  the  market  as  well.  The  two  sets  of 
change  percentages,  market  and  system,  are  then  brought  through  a  regression  analysis  to 
detennine  the  beta  coefficient.  Each  system  is  compared  to  the  market  to  determine  the 
beta. 


C.  TECHNOLOGY  LIFE  CYCLE 

The  technology  life  cycle  is  the  timeline  for  developing  technology  and 
recovering  the  costs  for  that  development  (Housel  &  Bergin,  2013).  The  digital 
environment  will  be  proactive  and  demand  responsiveness  from  IT  investments  (Chang 
&  West,  2006)  and  managers  need  to  invest  in  the  technology  life  cycle  to  meet  those 
challenges.  There  are  several  technology  life  cycle  models,  but  they  generally  start  with 
an  assessment  or  investigation,  then  move  to  a  design  or  selection  phase,  followed  by 
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implementation  or  production,  and  finally,  move  into  an  evaluation  and  analysis  phase 
before  starting  the  cycle  over  (Housel  &  Bergin,  2013).  Volatility  measurements  are 
concerned  operationally  with  the  management,  evaluation,  analysis  phase  but  can  be  used 
in  the  other  phases  of  the  technology  life  cycle  by  determining  how  the  volatility  from 
previous  systems  is  being  addressed. 

While  implementing  a  system,  the  value  of  IT  is  not  only  about  efficiency  but  also 
the  integration  of  organizations  and  users  (Yang  &  Melitski,  2007).  The  systems  should 
reach  the  most  customers  based  on  hardware  that  users  have  available  (Smith,  2007).  The 
system  should  link  computers  and  users  via  networks  and  allow  the  interaction  with  one 
another  to  produce  benefits  and  achieve  objectives  (Chang  &  West,  2006).  The 
interaction  and  growth  of  users  could  allow  an  IT  system  to  provide  significant  software 
and  computer  power  at  lower  price  points  while  providing  better  performance  at  lower 
prices  (Smith,  2007).  The  challenge  for  an  organization  is  to  produce  one-stop  service  for 
IT  systems  that  provide  cost  efficiencies  and  standardization  while  still  providing 
security,  privacy,  and  innovation  (Yang  &  Melitski,  2007). 

If  performance  of  an  IT  system  is  normally  distributed  then  the  volatility  of  the 
system  can  be  represented  by  the  standard  deviation,  with  one  standard  deviation 
representing  two-thirds  of  the  possible  outcomes  and  two  standard  deviations 
representing  95%  of  the  outcomes  (Copeland  &  Tufano,  2004).  The  performance  of  the 
IT  system  through  the  use  of  metrics  provides  a  nominal  input  by  which  to  calculate  the 
volatility.  The  volatility  then  becomes  a  metric  by  which  to  compare  systems  to  one 
another  and  determine  how  risk  is  reduced  and  the  system  becomes  more  predictable.  The 
volatility  may  also  assist  in  detennining  as  which  stage  of  the  technology  life  cycle  the 
system  resides  providing  managers  an  indication  of  what  IT  investments  to  make  and 
possibly  answer  which  IT  systems  provide  the  best  return  and  how  to  deploy  the  systems 
(Housel  and  Mun,  2013).  The  decisions  still  need  to  be  linked  to  the  IT  strategy  and 
budgets  while  balancing  stakeholders  competing  interests  and  demands  (Housel  & 
Bergin,  2013).  The  time  value  of  money  needs  to  be  considered,  as  benefits  may  not 
appear  for  some  time,  even  though  the  expenditure  may  be  required  today.  The  niche  area 
is  the  first  area  of  technology  adoption  followed  by  the  unregulated  spaces,  certified 
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applications,  recommended  practice,  and  finally  mandatory  practice  (Smith,  2007).  The 
volatility  measurement  will  have  a  similar  adoption  pattern,  by  first  proving  its  usefulness 
in  increasing  the  accuracy  of  forecast  in  a  niche  area. 
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IV.  DATA 


A.  AVERAGE  RETURNS 
1.  System  1 

The  rolling  daily  averages  for  System  1  (Figure  1)  shows  a  positive  trend  to 
period  310,  which  is  just  short  of  one  year,  then  stabilizing  in  the  number  of  reports 
generated  through  the  remainder  of  the  observed  time  periods.  The  60-day  moving 
average  is  not  a  stable  with  a  significant  increase  at  period  180  and  volatile  returns 
through  period  741  until  a  significant  drop. 


System  1  Daily  Averages 


Rolling  Average 


Moving  Average  (60 
days) 


Figure  1 .  System  1  Daily  Averages 


There  are  two  conclusions  observed  through  the  data.  The  first  is  the  significant 
increase  in  reports  to  period  180,  approximately  6  months,  which  is  indicative  of  the 
conquering  of  the  learning  curve.  The  second  conclusion  is  that  the  system  may  have 
reached  the  end  of  its  usefulness  with  the  drop  to  almost  zero  in  the  60-day  moving 
average.  The  weekly  and  monthly  (Figures  2  and  3)  averages  also  support  those 
conclusions. 
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System  1  Weekly 


Figure  2.  System  1  Weekly  Averages 


System  1  Monthly 


#  of  Reports  Monthly 

Rolling  Average 

Moving  Average  (60 
days) 


Figure  3.  System  1  Monthly  Averages 


2.  System  2 

For  system  2  (Figure  4),  the  rolling  average  shows  a  continued  positive  trend  with 
the  greatest  rate  of  change  occurring  to  the  month  8  point.  The  60-day  moving  average  is 
showing  a  stabilization  of  approximately  200  reports  per  month. 
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As  with  System  1,  the  large  increase  in  month  5  appears  to  indicate  a  learning 
curve  with  increased  readings  over  the  next  4  months.  The  stabilization  of  high  readings 
indicates  a  system  that  is  of  continued  benefit. 


System  2  Monthly 


#  of  Reports  Monthly 

•  Rolling  Average 

Moving  Average  (60 
days) 


Figure  4.  System  2  Monthly  Average 


System  2  is  one  of  two  systems  observed  that  have  identifiable  increments 
(Figures  5-8).  As  expected,  the  latest  increment  (Figure  8)  produces  the  greatest  number 
of  reports,  out-producing  all  of  the  other  increments  combined.  With  the  exception  of  a 
recent  sharp  increase  in  System  2.0  (Figure  5),  the  previous  increments  appear  to  be  at 
the  end  of  their  life  cycle.  Systems  2.1  and  2.2  (Figures  5  and  6)  appear  to  have  been 
mature  when  the  database  started  collecting  data  which  would  account  for  the  higher 
starting  averages  and  the  negative  trends  since  reporting  supports  the  end  of  life  cycle 
observation.  System  2.3  (Figure  8)  continues  to  show  positive  trends  in  both  its  rolling 
average  and  the  60-day  moving  average. 
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System  2.0  Monthly 


#  of  Reports  Monthly 

■Rolling  Average 

Moving  Average  (60 
days) 


Figure  5.  System  2.0  Monthly  Averages 


System  2.1  Monthly 


#  of  Reports  Monthly 

■  Rolling  Average 

Moving  Average  (60 
days) 


Figure  6.  System  2. 1  Monthly  Averages 
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System  2.2  Monthly 


#  of  Reports  Monthly 

Rolling  Average 

Moving  Average  (60 
days) 


Figure  7.  System  2.2  Monthly  Averages 


System  2.3  Monthly 


#  of  Reports  Monthly 

•  Rolling  Average 

Moving  Average  (60 
days) 


Figure  8.  System  2.3  Monthly  Averages 


3.  System  3 

Similar  to  System  2  (Figure  4),  System  3  (Figure  9)  shows  positive  trends  for  both 
the  rolling  average  60-day  moving  average.  Unlike  System  2,  the  learning  curve  for 
System  3  appears  to  occur  at  a  later  timeframe  of  16  months.  The  continued  increase  of 
readings  indicates  a  system  that  is  of  continued  benefit. 
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System  3  Monthly 


#  of  Reports  Monthly 
Rolling  Average 
Moving  Average  (60  days) 


Figure  9.  System  3  Monthly  Averages 


System  3  is  the  other  system  observed  that  has  identifiable  increments.  As  with 
System  2,  the  latest  increment  produces  the  greatest  number  of  reports.  Interestingly, 
there  was  also  a  recent  sharp  increase  in  System  3.0  (Figure  10);  otherwise  it  also 
appeared  to  be  at  the  end  of  their  life  cycle.  System  3.1  (Figure  11)  continues  to  show 
positive  trends  in  both  its  rolling  average  and  the  60-day  moving  average. 


System  3.0  Monthly 


#  of  Reports  Monthly 

Rolling  Average 

Moving  Average  (60 
days) 


Figure  10.  System  3.0  Monthly  Averages 
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System  3.1  Monthly 


#  of  Reports  Monthly 

Rolling  Average 

Moving  Average  (60 
days) 


Figure  1 1 .  System  3.1  Monthly  Averages 


4.  System  4 

For  System  4  (Figure  12),  the  rolling  average  shows  a  positive  trend  with  the 
greatest  rate  of  change  occurring  up  to  8 -month  point,  which  is  similar  to  System  2 
(Figure  4).  However,  after  month  10,  the  trend  becomes  slightly  negative.  The  60-day 
moving  average  is  volatile,  showing  a  sharp  increase  at  month  6,  peak  at  month  7  and 
then  decreases  to  month  13  until  another  sharp  increase  at  month  26.  As  with  System  2, 
the  increases  in  rolling  average  to  month  8  appears  to  indicate  a  learning  curve.  The 
stabilization  of  high  readings  indicates  a  system  that  is  of  continued  benefit. 
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System  4  Monthly 


#  of  Reports  Monthly 

Rolling  Average 

Moving  Average  (60 
days) 


1  3  5  7  9  11  13  15  17  19  21  23  25  27 

Figure  12.  System  4  Monthly  Averages 

5.  System  5 

For  System  5  (Figure  13),  the  rolling  average  shows  a  positive  trend  with  the 
greatest  rate  of  change  occurring  up  to  the  9  month  point,  which  is  similar  to  System  2 
(Figure  4)  and  4  (Figure  12).  The  60-day  moving  average  is  volatile  with  large  peaks  up 
to  month  21  then  substantial  decreases  with  a  post  month  9  low  at  month  25.  The 
increases  in  rolling  average  to  month  9  appear  to  indicate  a  learning  curve.  The  volatility 
of  readings  indicates  other  factors  in  play  than  just  the  information  system. 
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System  5  Monthly 


#  of  Reports  Monthly 

Rolling  Average 

Moving  Average  (60 
days) 


Figure  13.  System  5  Monthly  Averages 


B.  UNIT  COST 

1.  System  1 

The  unit  cost  for  System  1  (Figure  14)  over  the  observed  timeframe  shows  a 
positive  trend  (unit  cost  going  down)  until  daily  point  774  which  is  over  two  years  but 
then  increases  (negative  trend)  for  the  remainder  of  the  observed  period.  The  low  unit 
cost  calculation  for  System  lof  $5797  means  nothing  until  it  is  compared  with  other 
systems  or  the  overall  signal  reports  market.  Unfortunately,  the  unit  cost  for  System  1  is 
well  above  the  market  unit  cost  which  was  below  $5000  for  the  entirety  of  the  observed 
period. 
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2.  System  2 

The  unit  cost  for  System  2  (Figure  15)  over  the  observed  timeframe  shows  a 
positive  trend  (unit  cost  going  down)  throughout  the  observed  period.  This  indicates 
strength  in  benefit  returns  for  System  2. 
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The  increments  tell  a  different  story.  System  2.1  (Figure  16)  actually  shows  a 
positive  trend  for  the  majority  of  the  observed  period,  however  the  unit  cost  is  so  high 
that  it  indicates  a  system  that  costs  more  than  the  benefits  received  warrants  and  therefore 
should  be  a  system  considered  for  shut  down  or  replacement.  Unlike  System  2.1  (Figure 
16),  System  2.2  (Figure  17)  definitely  shows  a  negative  trend  and  indicates  a  system  that 
once  was  efficient  but  has  since  become  a  system  that  does  not  warrant  the  cost  and 
should  be  replaced.  System  2.3  (Figure  18)  shows  a  positive  trend  and  outweighs  all  the 
other  System  2  increments. 
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System  2.1  Unit  Cost 


Figure  16.  System  2.1  Unit  Cost 
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Figure  18.  System  2.3  Unit  Cost 


3.  System  3 

The  unit  cost  for  System  3  (Figure  19)  over  the  observed  timeframe  shows  a 
positive  trend  (unit  cost  going  down)  throughout  the  observed  period.  The  rapid  decrease 
in  unit  cost  is  more  in  line  with  the  market  average  indicating  a  maturing  of  the  system 
and  strength  in  benefit  returns  for  System  3. 
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Even  though  System  3.0  (Figure  20)  shows  a  positive  trend,  its  high  unit  cost 
indicates  a  system  that  does  not  warrant  the  costs  and  should  be  tenninated  as  opposed  to 
System  3.1  (Figure  21)  whose  unit  cost  is  84%  lower  than  System  3.0.  Incremental 
System  3.1  is  the  primary  driver  for  System  3. 
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4.  System  4 

System  4  (Figure  21)  shows  a  negative  trend  to  point  649  and  then  a  positive 
trend  for  the  remainder  of  the  observation  period.  This  would  be  an  example  of  a  system 
that  requires  further  research.  Unless  it  has  a  specific  mission  that  it  is  meeting  that  it 
should  be  considered  for  improvement  or  termination  as  the  amount  of  reports  generated 
does  not  warrant  the  assumed  cost. 


5.  System  5 

Similar  to  System  4  (Figure  21),  System  5  (Figure  22)  is  another  example  of  a 
system  that  requires  further  research.  Unless  it  has  a  specific  mission  that  it  is  meeting 
that  it  should  be  considered  for  improvement  or  termination  as  the  amount  of  reports 
generated  does  not  warrant  the  high  unit  cost.  The  positive  trend  through  the  observed 
period  only  results  in  a  low  of  over  $8000  for  its  unit  cost  which  is  well  above  the  market 
unit  cost. 
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6.  Market 

Figure  23  is  the  graphical  representation  of  all  the  systems.  The  market  (all 
systems)  line  shows  a  positive  trend  with  the  latest  reading  being  $2500.  Only  two 
systems  get  under  that  average,  Systems  2  and  3.  Systems  1,  4,  and  5  would  require 
further  research  to  determine  what  is  driving  their  unit  cost  to  be  above  average  however 
since  those  three  systems  have  unit  costs  3x  the  market  unit  cost,  they  should  be  reviewed 
with  the  purpose  of  removing  them. 
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Unit  Cost 


Figure  23.  Market  Unit  Cost  Comparisons 


C.  GARCH 

The  GARCH  forecasted  volatility  charts  are  located  in 
following  table  (Table  1)  provides  a  summary  of  the  results. 


All  Systems 
— System  1 
System2 

- System3 

System4 
- System5 


Appendix  A.  The 
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System 

Forecasted  Volatility 

Notes 

1 

12.95% 

2 

7.81% 

2.0 

20.72% 

2.1 

17% 

Decreasing  over  time 

2.2 

16.00% 

Slightly  increasing  over  time 

2.3 

10.67% 

3 

721% 

skewed  by  System  3.0 

3.0 

41.77% 

3.1 

21.09% 

4 

19.59% 

5 

21.53% 

Market 

7.38% 

Table  1.  Forecasted  Volatility 


As  expected  the  overall  market  volatility  and  forecasted  volatility  was  lower  with 
all  of  the  data  points  smoothing  out  the  results.  Unexpectedly,  that  was  not  the  case  with 
System  3  where  the  rapid  change  and  combination  of  System  3.0  and  3.1  resulted  in  even 
more  volatility.  As  with  the  unit  cost,  the  forecasted  volatility  measurements  point  to 
System  2,  specifically  incremental  System  2.3  as  the  system  of  choice. 

In  the  financial  sector,  the  inverse  relation  between  volatility  and  stock  market 
returns  is  well  known  (Lee  &  Rye,  2013);  the  data  shows  that  this  trend  also  holds  in 
returns  in  information  systems.  The  higher  volatility  systems  were  also  the  lower 
perfonning  systems,  although  the  volatility  should  be  measured  on  an  incremental  system 
basis  to  separate  the  underperforming  systems  from  the  perfonning  systems. 
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D.  BETA 

The  regression  analysis  results  are  located  in  Appendix  B.  The  following  table 
(Table  2)  provides  the  beta  for  each  system. 


System 

Beta 

1 

0.0098 

2 

0.0153 

2.0 

-0.0533 

2.1 

0.0969 

2.2 

0.1176 

2.3 

0.6040 

3 

0.2321 

3.0 

0.0461 

3.1 

0.2099 

4 

0.1485 

5 

0.0808 

Table  2.  System  Betas 


The  analysis  suggests  that  beta  may  be  a  better  indicator  of  successful  systems 
assuming  that  the  overall  market  of  systems  is  improving  which  is  the  case  for  this 
analysis.  Using  unit  cost  as  a  primary  factor  in  ranking  systems  then  Systems  2.3  and  3.1 
are  the  top  perfonners,  which  correlates  to  the  higher  beta  levels.  It  could  also  be  useful 
in  determining  the  systems  that  should  be  reviewed  for  termination  with  the  lowest  three 
betas  being  Systems  2.0,  1,  and  3.0  which  were  identified  in  the  average  returns  and  unit 
cost  sections  as  the  underperfonning  systems. 
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V.  CONCLUSION 


One  of  the  first  steps  is  to  ensure  that  you  are  measuring  precisely  the  intended 
values  (Yang  &  Melitski,  2007).  Management  needs  to  embrace  business  models  that 
assist  in  identifying  those  measures  including  agility  capacity  (Guertin  et.  ah,  2012). 
Whether  those  metrics  are  internal  versus  external  factors,  efficiency  versus 
effectiveness,  interagency  collaboration,  vertical  and  horizontal  integration,  or  distinct 
strategic  value  (Yang  &  Melitski,  2007),  volatility  measurements  assist  in  the  effort  by 
measuring  the  deviations  of  those  metrics.  Development,  clarification,  measurement,  and 
evaluation  of  all  competing  strategic  values  are  paramount  to  ensure  a  successful  IT 
system  implementation  (Yang  &  Melitski,  2007).  The  volatility  measurements  aids  in 
metric  evaluation  providing  the  additional  information  needed  to  compare  systems  and 
make  better  decisions. 

Demonstrating  the  competing  values  of  IT  systems  allows  the  IT  managers  to 
make  appropriate  comparisons;  however,  care  should  be  taken  in  generalizing  the  results 
as  other  factors  may  affect  those  values  (Yang  &  Melitski,  2007).  Volatility 
measurements  take  consideration  of  all  factors  and  give  a  baseline  from  which  the  IT 
manager  can  make  decisions  across  systems.  The  primary  factors  in  detennining 
volatility  in  IT  systems  are  the  amount  of  users,  interoperability  of  the  systems,  and  the 
volume  of  data  collection.  Metrics  are  important  for  decision  makers  as  they  can  be  used 
to  establish  goals  to  improve  efficiency  and  effectiveness,  are  measureable,  and  are  used 
to  assess  the  effectiveness  of  the  organization  and  its  leadership  (Kasunic,  2002).  The 
volatility  measurements  allow  systems  to  be  compared  appropriately  to  detennine  the 
gains  over  prior  iterations  along  with  aiding  in  detennining  which  options  to  exercise  for 
future  systems.  The  additional  capabilities  provided  by  volatility  measurements  will  go  a 
long  way  in  strengthening  IT  investments  and  the  perfonnance  review  of  those  systems. 
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A.  LIMITATIONS 

The  DoD  IEAs  engages  stakeholders  to  detennine  IT  transfonnation  including 
portfolio  assembly  and  implementation  (DoD  CIO,  2012).  Volatility  measurements  have 
shown  to  be  a  mechanism  by  which  those  stakeholders  can  optimize  the  portfolio 
assembly  and  implementation  of  infonnation  systems.  However,  there  are  limitations  for 
this  particular  research.  Human  rationality  and  intervention  are  the  most  valuable 
component  of  portfolio  assembly  (Long,  2010).  First,  the  incremental  systems  could 
potentially  skew  the  data  as  the  number  of  assets  available  for  the  different  systems  could 
have  changed  and  resulted  in  the  shifting  of  benefits  from  one  incremental  system  to 
another.  Second,  most  of  these  systems  are  brought  into  theatres  for  use  by  larger  assets 
such  U.S.  Navy  vessels  whose  numbers  and  operational  schedule  could  greatly  affect  the 
returns  of  those  systems.  Third,  the  unit  cost  data  only  takes  into  account  system  cost. 
The  fourth  limitation  is  that  this  research  assumes  that  all  reports  are  created  equal. 

B.  RESEARCH  QUESTIONS 

The  first  research  question,  do  volatility  measurements  provide  a  capability  to 
forecast  future  perfonnance  of  the  system,  is  answered  yes.  The  forecasted  volatility 
measurements  do  provide  a  forecast  of  future  perfonnance  as  shown  through  the  GARCH 
data.  The  lower  forecasted  volatility  systems  are  the  higher  performing  systems,  which  in 
this  research  was  System  2.3. 

The  second  research  question,  do  volatility  measurements  provide  timing  options 
for  the  implementation  of  incremental  or  new  infonnation  systems,  is  not  conclusive.  The 
period  of  observation  was  just  over  two  years  which  is  not  enough  observations  to  make  a 
conclusion.  The  two  year  timeframe  was  the  extent  of  the  data  within  the  database 
analyzed.  There  is  promise  that  volatility  measurements  will  provide  the  infonnation  for 
the  timing  of  options  as  more  data  is  gathered.  For  example,  the  ability  to  determine 
learning  curve  with  each  system  ranged  from  5  to  11  months  (System  2  and  3, 
respectively),  would  point  to  an  option  at  month  12  to  ensure  that  the  learning  curve  was 
passed  and  the  system  will  provide  expected  returns. 
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The  third  research  question,  can  beta  be  detennined  within  a  portfolio  of  systems 
based  on  volatility  measurements,  is  yes.  The  regression  analysis  results  that  provide  the 
beta  data  are  located  in  Appendix  B  with  the  table  of  beta  results  in  the  data  chapter. 
However,  future  research  of  the  use  of  beta  as  it  applies  to  infonnation  systems  is  needed. 
In  the  financial  industry,  beta  is  used  to  reduce  volatility/risk  of  a  portfolio  while 
maintaining  a  desired  return.  Having  market  data  of  only  five  systems  spanning  just  over 
a  two  year  timeframe  is  a  limiting  factor  for  this  research. 

C.  FUTURE  RESEARCH 

There  are  five  identified  areas  of  future  research  with  most  being  in  the  classified 
realm.  The  first  is  expanding  the  analysis  based  on  the  number  of  physical  systems  in  the 
U.S.  Navy  and  the  returns  they  generate.  Nonnalizing  the  data  based  on  those  numbers 
could  result  in  systems  that  appear  underperfonning  to  be  providing  substantial  results. 
The  second  is  similar  to  the  first  possibility,  but  concentrating  on  the  number  of  U.S. 
Navy  vessels  and  their  operational  schedules.  The  operational  schedule  is  a  major  factor 
in  the  return  of  these  systems.  The  third  area  identified  for  future  research  is  the  use  of 
actual  cost  data  to  provide  unit  cost.  The  costs  in  this  research  are  assumed  from  annual 
budget  submissions  as  actual  cost  data  would  push  the  research  to  a  classified  category. 
This  future  research  opportunity  can  also  be  combined  with  previous  research  from  Rios, 
Nelson,  Lambeth,  and  Clapp  and  provide  the  ROI  via  the  KVA  methodology.  The  fourth 
future  research  area  is  the  weighing  of  SIGINT  reports  to  determine  which  reports  are 
worth  more  or  less  than  others.  And  finally,  the  fifth  future  research  area  is  the  use  of 
beta  as  it  applies  to  infonnation  systems.  Previous  research  by  Nelson,  as  well  as  this 
research,  shows  that  beta  can  be  derived  but  actually  showing  the  use  of  the  beta  as  it 
relates  to  information  systems  needs  to  be  verified. 
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APPENDIX  A.  GARCH  VOLATILITY  LORECAST  RESULTS 


This  is  the  volatility  for  System  1  with  a  forecasted  volatility  of  12.95%. 


This  is  the  volatility  for  incremental  System  2.3  with  a  forecasted  volatility  of 
10.67%. 
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This  is  the  volatility  for  System  3.1  with  a  forecasted  volatility  of  21.09%. 


This  is  the  volatility  for  System  4  with  a  forecasted  volatility  of  19.59%. 
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This  is  the  volatility  for  System  5  with  a  forecasted  volatility  of  21.53%. 


This  is  the  volatility  for  all  the  systems  with  a  forecasted  volatility  of  7.38%. 
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APPENDIX  B.  REGRESSION  ANALYSIS  RESULTS 


System  1 


Regression  Results 

Intercept 

System  1 

Coefficients 

0.0548 

0.0098 

Standard  Error 

0.0082 

0.0045 

t-Statistic 

6.6990 

2.1594 

p-Value 

0.0000 

0.0344 

Lower  5% 

0.0385 

0.0007 

Upper  95% 

0.0712 

0.0189 
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System  2 


Regression  Results 

Intercept 

System  2 

Coefficients 

0.0486 

0.0153 

Standard  Error 

0.0090 

0.0077 

t-Statistic 

5.3697 

1.9894 

p-Value 

0.0000 

0.0507 

Lower  5% 

0.0305 

-0.0001 

Upper  95% 

0.0666 

0.0307 

System  2.3 


Regression  Results 

Intercept 

System  2.3 

Coefficients 

0.0042 

0.6040 

Standard  Error 

0.0048 

0.0172 

t-Statistic 

0.8784 

35.1954 

p-Value 

0.3802 

0.0000 

Lower  5% 

-0.0052 

0.5703 

Upper  95% 

0.0137 

0.6377 
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System  3 


Regression  Results 

Intercept 

System  3 

Coefficients 

0.0151 

0.2321 

Standard  Error 

0.0089 

0.0159 

t-Statistic 

1.7056 

14.6275 

p-Value 

0.0888 

0.0000 

Lower  5% 

-0.0023 

0.2009 

Upper  95% 

0.0326 

0.2633 

System  3.1 


Regression  Results 

Intercept 

System3.1 

Coefficients 

0.0282 

0.2099 

Standard  Error 

0.0097 

0.0236 

t-Statistic 

2.9111 

8.8904 

p-Value 

0.0039 

0.0000 

Lower  5% 

0.0091 

0.1634 

Upper  95% 

0.0472 

0.2563 
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System  4 


Regression  Results 

Intercept 

System  4 

Coefficients 

0.1118 

0.1485 

Standard  Error 

0.0070 

0.0154 

t-Statistic 

16.0066 

9.6403 

p-Value 

0.0000 

0.0000 

Lower  5% 

0.0981 

0.1182 

Upper  95% 

0.1256 

0.1788 

System  5 


Regression  Results 

Intercept 

System  5 

Coefficients 

0.0961 

0.0808 

Standard  Error 

0.0070 

0.0127 

t-Statistic 

13.7019 

6.3619 

p-Value 

0.0000 

0.0000 

Lower  5% 

0.0823 

0.0558 

Upper  95% 

0.1099 

0.1057 
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